Ordering of events between different input sources

ABSTRACT

A system and process for ensuring the smooth flow of electronic ink is described. Dynamic rendering is give priority over other event handlers. Priority may be the use of one or more queues to order when events occur and may be performing dynamic rendering prior to other steps.

RELATED APPLICATION INFORMATION

This application is a Continuation of U.S. application Ser. No.10/879,527 filed on Jun. 30, 2004; which is a Provisional of 60/513,591,filed Oct. 24, 2003, hereby incorporated by reference as to theirentirety.

This application claims priority to U.S. Ser. No. 60/513,591, filed Oct.24, 2003, entitled “Tablet Platform Controls and APIs” whose contentsare expressly incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Aspects of the present invention relate to image processing andinformation manipulation. More specifically, aspects of the presentinvention relate to receiving and handling electronic ink information.

2. Description of Related Art

Typical computer systems, especially computer systems using graphicaluser interface (GUI) systems such as Microsoft WINDOWS, are optimizedfor accepting user input from one or more discrete input devices such asa keyboard for entering text, and a pointing device such as a mouse withone or more buttons for driving the user interface.

The ubiquitous keyboard and mouse interface provides for fast creationand modification of documents, spreadsheets, database fields, drawings,photos and the like. However, there is a significant gap in theflexibility provided by the keyboard and mouse interface as comparedwith the non-computer (i.e., standard) pen and paper. With the standardpen and paper, a user edits a document, writes notes in a margin, anddraws pictures and other shapes and the like. In some instances, a usermay prefer to use a pen to mark-up a document rather than review thedocument on-screen because of the ability to freely make notes outsideof the confines of the keyboard and mouse interface.

When a user writes with a pen, he expects ink to flow from the pen tip.In the electronic ink realm, a similar goal exists. One issue thatimpedes electronic ink from flowing from a pen or stylus is how theelectronic ink is handled.

Previous approaches have attempted to handle all aspects of inking atthe same time. Once ink information exists, previous approaches haveattempted to render the ink, look for gestures, combine the inkinformation with other ink information, recognize the ink, and smooththe appearance of the ink, among other processes. These processes havecaused delays in displaying ink to a user, thereby making the user waitto enter more ink and, more importantly, reduce or eliminate theillusion that ink is flowing from the tip of the electronic stylus.

BRIEF SUMMARY

Aspects of the present invention address one or more of the issuesmentioned above, thereby providing flow of ink from a stylus. A dynamicrender object may be moved about in a process flow to reduce a delaybetween a user's movement and the display of electronic ink.

These and other aspects are addressed in relation to the Figures andrelated description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of a general-purpose digitalcomputing environment in which certain aspects of the present inventionmay be implemented;

FIGS. 1B through 1M show a general-purpose computer environmentsupporting one or more aspects of the present invention.

FIG. 2 shows a display for a stylus-based input system according toaspects of the present invention.

FIG. 3 shows a system for handling the flow of electronic ink inaccordance with aspects of the present invention.

FIG. 4 shows an alternative approach for handling the flow of electronicink in accordance with aspects of the present invention.

FIG. 5 shows an object mode that may support aspects of the presentinvention.

FIG. 6 shows a system for handling electronic ink in accordance withaspects of the present invention.

FIG. 7 shows a system for handling electronic ink in accordance withaspects of the present invention.

FIGS. 8A and 8B show various approaches for using an input queue andoutput queue in accordance with aspects of the present invention.

FIGS. 9 and 10 show methods for creating and using systems for handlingink in accordance with aspects of the present invention.

FIG. 11 shows a system with a protocol for alerting an ink collectionobject in accordance with aspects of the present invention

FIG. 12 shows, a system with an additional queue in accordance withaspects of the present invention.

FIG. 13 shows a system with separated real time stylus components inaccordance with aspects of the present invention.

FIG. 14 shows how deep a gesture recognizer may look for gestures inaccordance with aspects of the present invention.

FIG. 15 shows a system for exception handling in accordance with aspectsof the present invention.

FIG. 16 shows managed and unmanaged code in accordance with aspects ofthe present invention.

FIGS. 17A and 17B show how data may be placed in different locations ina queue to ensure synchronization in accordance with aspects of thepresent invention.

FIGS. 18 and 19 show various systems for handling pen data in accordancewith aspects of the present invention.

FIG. 20 shows a queue in accordance with aspects of the presentinvention.

DETAILED DESCRIPTION

Aspects of the present invention relate to handing stylus events in anefficient manner to quickly render electronic ink to a user.

This document is divided into sections to assist the reader. Thesesections include: characteristics of ink, terms, general-purposecomputing environment, real-time inking overview, object model, dynamicrendering and wet ink, gesture recognition, synchronous and asynchronousprocesses, cascading, dynamic plug-in collection modification, errorpropagation, managed/unmanaged illustrations, data sets and flows, datasynchronization, and application programming interfaces.

Characteristics of Ink

As known to users who use ink pens, physical ink (the kind laid down onpaper using a pen with an ink reservoir) may convey more informationthan a series of coordinates connected by line segments. For example,physical ink can reflect pen pressure (by the thickness of the ink), penangle (by the shape of the line or curve segments and the behavior ofthe ink around discreet points), and the speed of the nib of the pen (bythe straightness, line width, and line width changes over the course ofa line or curve). Further examples include the way ink is absorbed intothe fibers of paper or other surface it is deposited on. These subtlecharacteristics also aid in conveying the above listed properties.Because of these additional properties, emotion, personality, emphasisand so forth can be more instantaneously conveyed than with uniform linewidth between points.

Electronic ink (or ink) relates to the capture and display of electronicinformation captured when a user uses a stylus-based input device.Electronic ink refers to a sequence or any arbitrary collection ofstrokes, where each stroke is comprised of a sequence of points. Thestrokes may have been drawn or collected at the same time or may havebeen drawn or collected at independent times and locations and forindependent reasons. The points may be represented using a variety ofknown techniques including Cartesian coordinates (X, Y), polarcoordinates (r, Θ), and other techniques as known in the art. Electronicink may include representations of properties of real ink includingpressure, angle, speed, color, stylus size, and ink opacity. Electronicink may further include other properties including the order of how inkwas deposited on a page (a raster pattern of left to right then down formost western languages), a timestamp (indicating when the ink wasdeposited), indication of the author of the ink, and the originatingdevice (at least one of an identification of a machine upon which theink was drawn or an identification of the pen used to deposit the ink)among other information.

Terms

-   Ink A sequence or set of strokes with properties. A sequence of    strokes may include strokes in an ordered form. The sequence may be    ordered by the time captured or by where the strokes appear on a    page or in collaborative situations by the author of the ink. Other    orders are possible. A set of strokes may include sequences of    strokes or unordered strokes or any combination thereof. Further,    some properties may be unique to each stroke or point in the stroke    (for example, pressure, speed, angle, and the like). These    properties may be stored at the stroke or point level, and not at    the ink level-   Ink object A data structure storing ink with or without properties.-   Stroke A sequence or set of captured points. For example, when    rendered, the sequence of points may be connected with lines.    Alternatively, the stroke may be represented as a point and a vector    in the direction of the next point. In short, a stroke is intended    to encompass any representation of points or segments relating to    ink, irrespective of the underlying representation of points and/or    what connects the points.-   Point Information defining a location in space. For example, the    points may be defined relative to a capturing space (for example,    points on a digitizer), a virtual ink space (the coordinates in a    space into which captured ink is placed), and/or display space (the    points or pixels of a display device).-   Document Any electronic file that has a viewable representation and    content. A document may include a web page, a word processing    document, a note page or pad, a spreadsheet, a visual presentation,    a database record, image files, and combinations thereof.-   RealTimeStylus The real time stylus is an object that provides real    time stylus events on a given window handle within a given window    input rectangle. The real time styles can also be considered as a    framework to which to add plug-in objects that handle additional    functions. The plug-in objects may be added and removed as the    desired functionality of the real time stylus changes. The API for    the real time stylus may be referred to as RTS. RTS may be a core    API component to which developers can tie other functions. The RTS    packages the raw data coming from pen services and forwards the data    to the first plug-in object (if any). The real time stylus may have    one or more interfaces. In the case of two interfaces, these may be    a synchronous interface and an asynchronous interface. These    interfaces provide locations to which plug-ins may be connected to    the real time stylus These interfaces are for illustrative purposes    only. Other interfaces may be used as well.-   Real Time Ink Real time ink is the illusion that ink is flowing from    the tip of the stylus. Prior approaches have attempted to couple    significant processing steps with the rendering of ink, thereby    slowing down the display of newly received strokes. Real time ink    attempts to push the newly received strokes to the display as they    are received and attempts to separate the real-time processing steps    and the real-time display to operate together, thereby displaying    ink faster to a user.-   Plug-in A plug-in is a functional component that may be added to the    real time stylus object. If a plug-in is attached to a synchronous    interface of the real time stylus object, it may be referred to as a    synchronous plug-in. If a plug-in is attached to an asynchronous    interface of the real time stylus object, it may be referred to as    an asynchronous plug-in.-   Pen Service A system service component that interfaces with the    digitizer hardware driver and provides raw “packet” data that has    been pre-interpreted into standard “packet” and related structures,    along with other notifications independent of ink collection that    are still relevant to Tablet functionality (e.g.    TabletAdded/Removed, StylusButtonUp/Down, etc). Pen services provide    the primary mechanism for packets to be handled by a plug-in    collection.-   Plug-in Collection A collection of plug-ins, in one or more groups,    that are attached to the RTS. Where two collections exist, they may    be associated with the synchronous and asynchronous interfaces of    the RTS, respectively. Each collection may be executed in order.    Where two or more collections exist, they may be executed    independent of each other because they may be attached to different    interfaces of the RTS (or RTSs). Data returns to the RTS after being    handled by each plug-in. The order of plug-ins in the collection may    affect the output of the collection.-   Plug-in Chain A chain of plug-ins where each grouping of plug-ins is    connected in series. Here, data work their way through each plug-in    in the chain prior to returning to the RTS. The chain may also be    referred to as a “daisy chain” of plug-ins. The order of the    plug-ins in the chain or daisy chain may affect the output of the    chain.-   Input Queue A temporary holding queue for packets or objects that    are generated by a synchronous plug-in collection or plug-in chain    for reprocessing by the synchronous plug-in collection or plug-in    chain. In short, the synchronous plug-in collection or plug-in chain    can push objects into the input queue.-   Dynamic Rendering The incremental process of drawing ink while the    pen is touching the screen. As the pen moves across the digitizer a    trail of “ink” is left behind on the screen. The effect appears as    if the ink is flowing from the pen as it is being drawn. The ink is    drawn piecemeal as the data is received. The rendering may also have    additional effects applied such as anti-aliasing and the like.-   Static Rendering The process of rendering an entire ink stroke at    one time. The data is already available before the ink is drawn and    the entire stroke is drawn as a single unit. Static rendering may    also have additional effects applied such as Bezier smoothing and    anti-aliasing.-   Dynamic Renderer The developer may optionally instantiate a dynamic    renderer to automatically provide highly efficient rendering of real    time packet data to a GDI rendering surface. Dynamic renderers for    other rendering surfaces, like DirectX or a plotter are possible    using the same interface definition as the GDI-centric    DynamicRenderer provided as part of the implementation of aspects of    this invention.-   Gesture Recognizer The developer may optionally instantiate a    gesture recognizer to perform real-time recognition of strokes and    report when one or more strokes constitute a gesture in which the    developer has expressed interest. The gesture recognizer may, if it    is used, be placed in the synchronous or asynchronous collection or    chain.-   Output Queue Once the packet data has flowed through the collection    or chain of objects (and potentially has been modified), it is    deposited in the output queue where the RTS the sends it to the    collection of asynchronous plug-ins or chain.-   InkCollectingObject Describes an asynchronous plug-in object that    accumulates and stores ink data provided by the Real Time Stylus.-   ICO An ink collection object that may be in an asynchronous    collection or chain. It receives data from the output queue.    General-Purpose Computing Environment

FIG. 1A illustrates an example of a suitable computing systemenvironment 100 on which the invention may be implemented. The computingsystem environment 100 is only one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the invention. Neither should thecomputing system environment 100 be interpreted as having any dependencyor requirement relating to any one or combination of componentsillustrated in the exemplary computing system environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices including smart phones, smart watches, and personaldata assistants, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1A, an exemplary system for implementing theinvention includes a general-purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, random access memory(RAM), read only memory (ROM), electronically erasable programmable readonly memory (EEPROM), flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can accessed by computer 110.Communication media typically embodies computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of the anyof the above should also be included within the scope of computerreadable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as ROM 131 and RAM 132. A basicinput/output system 133 (BIOS), containing the basic routines that helpto transfer information between elements within computer 110, such asduring start-up, is typically stored in ROM 131. RAM 132 typicallycontains data and/or program modules that are immediately accessible toand/or presently being operated on by processing unit 120. By way ofexample, and not limitation, FIG. 1A illustrates operating system 134,application programs 135, other program modules 136, and program data137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1A illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disc drive 155 that reads from or writes to a removable,nonvolatile optical disc 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disc drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1A, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1A, for example, hard disk drive 141 isillustrated as storing operating system 144, application programs 145,other program modules 146, and program data 147. Note that thesecomponents can either be the same as or different from operating system134, application programs 135, other program modules 136, and programdata 137. Operating system 144, application programs 145, other programmodules 146, and program data 147 are given different numbers here toillustrate that, at a minimum, they are different copies. A user mayenter commands and information into the computer 110 through inputdevices such as a digital camera 163, a keyboard 162, and pointingdevice 161, commonly referred to as a mouse, trackball or touch pad.Other input devices (not shown) may include a microphone, joystick, gamepad, satellite dish, scanner, or the like. These and other input devicesare often connected to the processing unit 120 through a user inputinterface 160 that is coupled to the system bus 121, but may beconnected by other interface and bus structures, such as a parallelport, game port or a universal serial bus (USB). A monitor 191 or othertype of display device is also connected to the system bus 121 via aninterface, such as a video interface 190. In addition to the monitor,computers may also include other peripheral output devices such asspeakers 197 and printer 196, which may be connected through an outputperipheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1A. The logical connections depicted in FIG. 1Ainclude a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1A illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

It will be appreciated that the network connections shown are exemplaryand other means of establishing a communications link between thecomputers can be used. The existence of any of various well-knownprotocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed,and the system can be operated in a client-server configuration topermit a user to retrieve web pages from a web-based server. Any ofvarious conventional web browsers can be used to display and manipulatedata on web pages.

A programming interface (or more simply, interface) may be viewed as anymechanism, process, protocol for enabling one or more segment(s) of codeto communicate with or access the functionality provided by one or moreother segment(s) of code. Alternatively, a programming interface may beviewed as one or more mechanism(s), method(s), function call(s),module(s), object(s), etc. of a component of a system capable ofcommunicative coupling to one or more mechanism(s), method(s), functioncall(s), module(s), etc. of other component(s). The term “segment ofcode” in the preceding sentence is intended to include one or moreinstructions or lines of code, and includes, e.g., code modules,objects, subroutines, functions, and so on, regardless of theterminology applied or whether the code segments are separatelycompiled, or whether the code segments are provided as source,intermediate, or object code, whether the code segments are utilized ina runtime system or process, or whether they are located on the same ordifferent machines or distributed across multiple machines, or whetherthe functionality represented by the segments of code are implementedwholly in software, wholly in hardware, or a combination of hardware andsoftware.

Notionally, a programming interface may be viewed generically, as shownin FIG. 1B or FIG. 1C. FIG. 1B illustrates an interface Interface1 as aconduit through which first and second code segments communicate. FIG.1C illustrates an interface as comprising interface objects I1 and I2(which may or may not be part of the first and second code segments),which enable first and second code segments of a system to communicatevia medium M. In the view of FIG. 1C, one may consider interface objectsI1 and I2 as separate interfaces of the same system and one may alsoconsider that objects I1 and I2 plus medium M comprise the interface.Although FIGS. 1B and 1C show bi-directional flow and interfaces on eachside of the flow, certain implementations may only have information flowin one direction (or no information flow as described below) or may onlyhave an interface object on one side. By way of example, and notlimitation, terms such as application programming interface (API), entrypoint, method, function, subroutine, remote procedure call, andcomponent object model (COM) interface, are encompassed within thedefinition of programming interface.

Aspects of such a programming interface may include the method wherebythe first code segment transmits information (where “information” isused in its broadest sense and includes data, commands, requests, etc.)to the second code segment; the method whereby the second code segmentreceives the information; and the structure, sequence, syntax,organization, schema, timing and content of the information. In thisregard, the underlying transport medium itself may be unimportant to theoperation of the interface, whether the medium be wired or wireless, ora combination of both, as long as the information is transported in themanner defined by the interface. In certain situations, information maynot be passed in one or both directions in the conventional sense, asthe information transfer may be either via another mechanism (e.g.information placed in a buffer, file, etc. separate from informationflow between the code segments) or non-existent, as when one codesegment simply accesses functionality performed by a second codesegment. Any or all of these aspects may be important in a givensituation, e.g., depending on whether the code segments are part of asystem in a loosely coupled or tightly coupled configuration, and sothis list should be considered illustrative and non-limiting.

This notion of a programming interface is known to those skilled in theart and is clear from the foregoing detailed description of theinvention. There are, however, other ways to implement a programminginterface, and, unless expressly excluded, these too are intended to beencompassed by the claims set forth at the end of this specification.Such other ways may appear to be more sophisticated or complex than thesimplistic view of FIGS. 1B and 1C, but they nonetheless perform asimilar function to accomplish the same overall result. We will nowbriefly describe some illustrative alternative implementations of aprogramming interface.

A. Factoring

A communication from one code segment to another may be accomplishedindirectly by breaking the communication into multiple discretecommunications. This is depicted schematically in FIGS. 1D and 1E. Asshown, some interfaces can be described in terms of divisible sets offunctionality. Thus, the interface functionality of FIGS. 1B and 1C maybe factored to achieve the same result, just as one may mathematicallyprovide 24, or 2 times 2 times 3 times 2. Accordingly, as illustrated inFIG. 1D, the function provided by interface Interface1 may be subdividedto convert the communications of the interface into multiple interfacesInterface1A, Interface1B, Interface1C, etc. while achieving the sameresult. As illustrated in FIG. 1E, the function provided by interface I1may be subdivided into multiple interfaces I1 a, I1 b, I1 c, etc. whileachieving the same result. Similarly, interface I2 of the second codesegment which receives information from the first code segment may befactored into multiple interfaces I2 a, I2 b, I2 c, etc. When factoring,the number of interfaces included with the 1st code segment need notmatch the number of interfaces included with the 2nd code segment. Ineither of the cases of FIGS. 1D and 1E, the functional spirit ofinterfaces Interface1 and I1 remain the same as with FIGS. 1B and 1C,respectively. The factoring of interfaces may also follow associative,commutative, and other mathematical properties such that the factoringmay be difficult to recognize. For instance, ordering of operations maybe unimportant, and consequently, a function carried out by an interfacemay be carried out well in advance of reaching the interface, by anotherpiece of code or interface, or performed by a separate component of thesystem. Moreover, one of ordinary skill in the programming arts canappreciate that there are a variety of ways of making different functioncalls that achieve the same result.

B. Redefinition

In some cases, it may be possible to ignore, add or redefine certainaspects (e.g., parameters) of a programming interface while stillaccomplishing the intended result. This is illustrated in FIGS. 1F and1G. For example, assume interface Interface1 of FIG. 1B includes afunction call Square (input, precision, output), a call that includesthree parameters, input, precision and output, and which is issued fromthe 1st Code Segment to the 2nd Code Segment. If the middle parameterprecision is of no concern in a given scenario, as shown in FIG. 1F, itcould just as well be ignored or even replaced with a meaningless (inthis situation) parameter. One may also add an additional parameter ofno concern. In either event, the functionality of square can beachieved, so long as output is returned after input is squared by thesecond code segment. Precision may very well be a meaningful parameterto some downstream or other portion of the computing system; however,once it is recognized that precision is not necessary for the narrowpurpose of calculating the square, it may be replaced or ignored. Forexample, instead of passing a valid precision value, a meaningless valuesuch as a birth date could be passed without adversely affecting theresult. Similarly, as shown in FIG. 1G, interface I1 is replaced byinterface I1′, redefined to ignore or add parameters to the interface.Interface I2 may similarly be redefined as interface I2′, redefined toignore unnecessary parameters, or parameters that may be processedelsewhere. The point here is that in some cases a programming interfacemay include aspects, such as parameters, which are not needed for somepurpose, and so they may be ignored or redefined, or processed elsewherefor other purposes.

C. Inline Coding

It may also be feasible to merge some or all of the functionality of twoseparate code modules such that the “interface” between them changesform. For example, the functionality of FIGS. 1B and 1C may be convertedto the functionality of FIGS. 1H and 1I, respectively. In FIG. 1H, theprevious 1st and 2nd Code Segments of FIG. 1B are merged into a modulecontaining both of them. In this case, the code segments may still becommunicating with each other but the interface may be adapted to a formwhich is more suitable to the single module. Thus, for example, formalCall and Return statements may no longer be necessary, but similarprocessing or response(s) pursuant to interface Interface1 may still bein effect. Similarly, shown in FIG. 1I, part (or all) of interface I2from FIG. 1C may be written inline into interface I1 to form interfaceI1″. As illustrated, interface I2 is divided into I2 a and I2 b, andinterface portion I2 a has been coded in-line with interface I1 to forminterface I1″. For a concrete example, consider that the interface I1from FIG. 1C performs a function call square (input, output), which isreceived by interface I2 which after processing the value passed withinput (to calculate the square of an input) by the second code segment,passes back the squared result with output. In such a case, theprocessing performed by the second code segment (squaring input) can beperformed by the first code segment without a call to the interface.

D. Divorce

A communication from one code segment to another may be accomplishedindirectly by breaking the communication into multiple discretecommunications. This is depicted schematically in FIGS. 1J and 1K. Asshown in FIG. 1J, one or more piece(s) of code (Divorce Interface(s),since they divorce functionality and/or interface functions from theoriginal interface) are provided to convert the communications on thefirst interface, Interface1, to conform them to a different interface,in this case interfaces Interface2A, Interface2B and Interface2C. Thismight be done, e.g., where there is an installed base of applicationsdesigned to communicate with, say, an operating system in accordancewith an Interface1 protocol, but then the operating system is changed touse a different interface, in this case interfaces Interface2A,Interface2B and Interface2C. The point is that the original interfaceused by the 2nd Code Segment is changed such that it is no longercompatible with the interface used by the 1st Code Segment, and so anintermediary is used to make the old and new interfaces compatible.Similarly, as shown in FIG. 1K, a third code segment can be introducedwith divorce interface DI1 to receive the communications from interfaceI1 and with divorce interface DI2 to transmit the interfacefunctionality to, for example, interfaces I2 a and I2 b, redesigned towork with DI2, but to provide the same functional result. Similarly, DI1and DI2 may work together to translate the functionality of interfacesI1 and I2 of FIG. 1C to a new operating system, while providing the sameor similar functional result.

E. Rewriting

Yet another possible variant is to dynamically rewrite the code toreplace the interface functionality with something else but whichachieves the same overall result. For example, there may be a system inwhich a code segment presented in an intermediate language (e.g.Microsoft IL, Java ByteCode, etc.) is provided to a Just-in-Time (JIT)compiler or interpreter in an execution environment (such as thatprovided by the Net framework, the Java runtime environment, or othersimilar runtime type environments). The JIT compiler may be written soas to dynamically convert the communications from the 1st Code Segmentto the 2nd Code Segment, i.e., to conform them to a different interfaceas may be required by the 2nd Code Segment (either the original or adifferent 2nd Code Segment). This is depicted in FIGS. 1L and 1M. As canbe seen in FIG. 1L, this approach is similar to the Divorce scenariodescribed above. It might be done, e.g., where an installed base ofapplications are designed to communicate with an operating system inaccordance with an Interface1 protocol, but then the operating system ischanged to use a different interface. The JIT Compiler could be used toconform the communications on the fly from the installed-baseapplications to the new interface of the operating system. As depictedin FIG. 1M, this approach of dynamically rewriting the interface(s) maybe applied to dynamically factor, or otherwise alter the interface(s) aswell.

It is also noted that the above-described scenarios for achieving thesame or similar result as an interface via alternative embodiments mayalso be combined in various ways, serially and/or in parallel, or withother intervening code. Thus, the alternative embodiments presentedabove are not mutually exclusive and may be mixed, matched and combinedto produce the same or equivalent scenarios to the generic scenariospresented in FIGS. 1B and 1C. It is also noted that, as with mostprogramming constructs, there are other similar ways of achieving thesame or similar functionality of an interface which may not be describedherein, but nonetheless are represented by the spirit and scope of theinvention, i.e., it is noted that it is at least partly thefunctionality represented by, and the advantageous results enabled by,an interface that underlie the value of an interface.

FIG. 2 illustrates an illustrative tablet PC 201 that can be used inaccordance with various aspects of the present invention. Any or all ofthe features, subsystems, and functions in the system of FIG. 1 can beincluded in the computer of FIG. 2. Tablet PC 201 includes a largedisplay surface 202, e.g., a digitizing flat panel display, preferably,a liquid crystal display (LCD) screen, on which a plurality of windows203 is displayed. Using stylus 204, a user can select, highlight, and/orwrite on the digitizing display surface 202. Examples of suitabledigitizing display surfaces 202 include electromagnetic pen digitizers,such as Mutoh or Wacom pen digitizers. Other types of pen digitizers,e.g., optical digitizers, may also be used. Tablet PC 201 interpretsgestures made using stylus 204 in order to manipulate data, enter text,create drawings, and/or execute conventional computer application taskssuch as spreadsheets, word processing programs, and the like.

The stylus 204 may be equipped with one or more buttons or otherfeatures to augment its selection capabilities. In one embodiment, thestylus 204 could be implemented as a “pencil” or “pen”, in which one endconstitutes a writing portion and the other end constitutes an “eraser”end, and which, when moved across the display, indicates portions of thedisplay are to be erased. Other types of input devices, such as a mouse,trackball, or the like could be used. Additionally, a user's own fingercould be the stylus 204 and used for selecting or indicating portions ofthe displayed image on a touch-sensitive or proximity-sensitive display.Consequently, the term “user input device”, as used herein, is intendedto have a broad definition and encompasses many variations on well-knowninput devices such as stylus 204. Region 205 shows a feedback region orcontact region permitting the user to determine where the stylus 204 ascontacted the display surface 202.

In various embodiments, the system provides an ink platform as a set ofCOM (component object model) services that an application can use tocapture, manipulate, and store ink. One service enables an applicationto read and write ink using the disclosed representations of ink. Theink platform may also include a mark-up language including a languagelike the extensible markup language (XML). Further, the system may useDCOM as another implementation. Yet further implementations may be usedincluding the Win32 programming model and the Net programming model fromMicrosoft Corporation.

Real-Time Inking Overview

Using a stylus or pen, a user creates electronic ink. The ink is handledby a system that allows the ink to be displayed closer to a time theuser creates the ink, rather than waiting until additional processeshave been completed.

FIG. 3 shows an illustrative system for handling electronic ink inaccordance with aspects of the present invention. An input managerreceives ink in the input manager 301. Various input managers exist inthe art including pens and digitizing tablets, Wacom tablets, and thelike. The existence of the ink data is referred to as a stylus inputevent 302. Next, the stylus input event 302 is handled by an inkgatherer 303. The ink collection object 303 performs initial handling ofthe information from the input manager 301. The system then dynamicallyrenders the ink 304 for output on display 305. Other components mayhandle the more complete processing of ink. For instance, a stroke maybe added to an existing ink object 306 (or may create a new ink objectto contain the ink stroke) and may associate the received ink (referredto as data) with one or more properties. This is shown by the componentwith ink object property 307. The ink object may then be redrawn (ifsmoothing is desired) 308 for display on display 305.

FIG. 4 shows an alternative approach to that of FIG. 3. FIG. 4 includesan internal stylus input source 401, an input manager 402 (which may ormay not have an input queue), an ink collection object 403, and elementwith ink object property 405, a real time inking object 404 and adisplay 406.

Two types of sets of inputs may be handled: data resulting from contactbetween a stylus and a digitizer and data resulting from movement madeabove a digitizer. The movements made above the digitizer that do notcontact the digitizer are referred to as “in-air stylus” inputs. Theinternal stylus input source separates the two sets of inputs and routesthem accordingly. The following lists various actions that occur in FIG.4:

-   -   A) An in-air stylus input event is added to an input manager        402's input queue.    -   B) The input manager 402 dispatches an in air stylus input to        ink collection 403.    -   C) The input manager 402 also outputs the in-air stylus input        for processing to determine if a focus change has occurred. The        real time inking object 404 (also referred to as a real time        stylus control) is also notified. The real time inking object        404 may request any needed data (ink color, etc.).    -   D) The in-air stylus event continues normal processing and        continues to the element with ink object property 405.    -   E) Stylus “down” events are received and sent to the real time        inking object 405.    -   F) Real time inking object 405 draws points as they are        received. This may be referred to as dynamic rendering.    -   G) Real time inking object 405 forwards the drawn events to the        input manager 402 to be dispatched at some later time as user        interface threads becomes available.    -   H) A stroke is collected and then added to an ink object.    -   I) The element first notifies the real time inking object 405 to        remove the dynamically drawn stroke, and then redraws the new        stroke. This operation may occur based on a stroke by stroke        basis or may apply on ink strokes.    -   J) The ink is rendered after all inking is complete.

The above approach as shown in FIG. 4 provides the benefits that ink lagis only critical for “stylus down” events. Ink lag is only perceptiblewhile the stylus is actively drawing. All other stylus events can bedelayed with little negative user impact.

The approach of FIG. 4 provide for multi-thread awareness beinglocalized in the input manager queue and the RTI object. It alsoguarantees that once focus has been established there will be no lag ordelay.

Object Model

FIG. 5 shows various objects that may be used in combination withaspects of the present invention including five parts that may becombined to various degrees. The following lists various terms that areused in the following description:

The first part is a pen services application 501 that supports thegathering of electronic ink. An example is wisptis.exe provided from theMicrosoft Corporation and used in Windows XP Tablet Edition.

Second, a real time stylus service 503 (shown associated with process 1502) is an application that forwards stylus data from pen services 501to the appropriate windows for collection. The real time stylus service503 may handle an unlimited number of objects or may be limited tominimize over usage. For instance, if limited, an illustrative number ofobjects per thread may be 16, 32, 64, 128, and the like. Of course,other values may be used.

Third, real time styluses 504 and 505 are shown in process 1. Real timestyluses 504 and 505 may receive stylus data from real time stylusservice 503. Each real time stylus object may be receiving stylus datafor a given section of a window or region (based on an associated windowor region for that real time stylus object.

To show how multiple processes may be implemented at the same time,process 2 506 is also shown. Real time stylus service 507 may alsoreceive stylus data from pen services 501 and forwards this informationto real time styluses 508 and 509.

The real time stylus objects 504, 505, 508 and 509 are shown in greaterdetail in FIG. 6. The pen services component 601 forwards data to thereal time stylus service 602 in data flow A. Next, the real time stylusservice 602 forwards stylus data onto one or more real time styluscomponents 603 in data flow B. Alternatively, as shown by the real timestylus service 602 shown by dashed lines, this service may be omittedand stylus data flowing directly to the real time stylus component 603.

The stylus data received at the real time stylus component 603 may bedirectly fed to pluggable components 606-608. Alternatively, receivedstylus data may be fed into its input queue 604 for ordered processing.The real time stylus component 603 next forwards stylus data to one ormore pluggable components. These components 606-608 may include adynamic renderer 608 with a visual store that stores the currentlyrendered stylus data. The data flows include flows C and D.

The dynamic renderer 608 may accumulate and store packet data for one ormore strokes in visual store 609. This is advantageous for instance whena stroke extends beyond an edge then reenters an inkable region whilethe stylus is still down. If the displayed window receives a repaintrequest while the pen is down, the store 609 provides the information torepaint the window quickly. If the dynamic renderer did not store stylusdata in 609, the system would have to wait to repaint the window andrender the most current stroke or strokes until the data made it out ofthe output queue and into the ink collection object 611.

The pluggable components 606-608 are class objects that may respond tocertain calls through an interface to process stylus data. One exampleof an interface that may be used includes IStylusSyncPlugin. Thecombination of pluggable components 606-608 allows a developer to fullycontrol, modify, and even delete data from the packet stream within theplug-ins. The modified packet stream from data flow E is then stored inthe output queue 605.

The output of output queue 605 flows through data flows F and G toanother collection of pluggable components 610 and 611. These componentsmay include a gesture recognizer 610 and an ink collection object 611with ink storage 612. A variety of additional plug-in objects may betapped by data flows F and G.

The decision to separate pluggable components 606-608 from 610 and 611may be based on a variety of criteria. For instance, components 606-608may be synchronous plug-ins and components 610 and 611 maybeasynchronous plug-ins. Alternatively, plug-ins with shorter stylus datahanding times may be handled by data flows C/D and longer stylus datahandling times addressed by data flows F/G. The components from onethread C/D may be exchanged with those of thread F/G.

One advantage of separating the two sets of pluggable components 606-608and 610-611 is that the pluggable components are handled by differentthreads. Here, the difference between the synchronous plug-ins and theasynchronous plug-ins lie in the thread they execute in, and the callingsequence (synchronous plug-ins may be called by the thread on which thereal time stylus 603 is executing and the asynchronous plug-ins arecalled by normally the user interface/application thread after thepacket stream has been processed by the synchronous plug-ins and storedin the output queue 605).

In some cases, there may be public communication from the real timestylus component 603 back to pen services 601 or real time stylusservice 602. In other cases, there is no public communication from thereal time stylus component 603 back to pen services 601 or real timestylus service 602. Preventing communication may help the flow of datafrom these components.

In some situations, the real time stylus component 603 may notify theplug-ins when retrieving data by sending calls to the plug-ins in apredefined sequence. The sequence and types of plug-ins that receivethese notifications may be controlled by the developer. The packet datain the events may be modified by any of these plug-ins 606-608 and610-611.

Aspects of the present invention work with a variety of data typesincluding packet data from pen services, notifications of changesregarding a display, tablet, pen, and the like, and other data sets thatmay be handled by the real time stylus. While the following descriptiondescribes the use of packet data from pen services, this is but one ofthe many types of data that may be used with the real time stylus 603.For the following, packet data is used as an illustrative example forthe data type handled by the RTS but is to be understood as referencingthe more general data that may be handled by the RTS.

The real time stylus component 603 may also include queues 604 and 605.The output queue 605 may maintain all of the packet data that the timestylus component 603 processes. Once the packet data has returned fromthe plug-in the packet data is added to the output queue 605 from dataflow E. The output queue 605 may then be used by the plug-ins (forinstance, asynchronous and generally including the ink collection object611). This may occur by extracting data out (data flow F) and buildingan ink object from the data held within in the output queue 605 in dataflow G.

The size of output queue 605 may or may not be fixed. If fixed, afterthe queue 605 is full, all subsequently received data packets may belost. If not fixed, the size of queue 605 may expand to receiveadditional data packets. One benefit of keeping a fixed size queue isthat it limits the backlog of data to that what can be processed in areasonable period of time. For instance, if the end-user is interactingwith the system and it becomes unresponsive, it is typical for theend-user to pause until the system is once again responsive, therebyallowing the queue to process the backlog without losing data. Also, if,for some reason, a high volume of stylus data was created, queue 605 mayhelp eliminate some of the data by having a fixed size.

The output queue 605 may be fixed in receiving data in order.Alternatively, as described below, data may be placed in output queue605 out of sequence to maintain synchronization with real timerendering.

Input queue 604 receives information in data flow B (or if no real timestylus service 602, then data flow A) The input queue provides a processto input data to plug-ins 606-608. Alternatively, stylus data may flowdirectly to plug-ins 606-608. One advantage to having input queue 604 asan intermediary between data flows B and C (here as data flow Z) is thatit allows created stylus data to be inserted where none exists.

The following describes a high level control flow.

-   -   a. A real time stylus 603 concludes processing packet data as        passed through plug-ins 606-608.    -   b. The real time stylus 603 stores the processed packet data in        the output queue 605.    -   c. The real time stylus 603 looks of there are any pending        packet data in the input queue 604. If there are data, then that        pack data from the input queue is picked up and processed in        step a above.    -   d. The real time stylus 603 looks if there are any pending        packet data in pen services 601. If yes, then the packet data        from pen services 601 is picked up and processed in step a        above.    -   e. This process repeats at step c.

The following describes an alternative approach to how the real timestylus object may handle the tablet pen data. It is appreciated that thefollowing may be applied with minor modification to FIGS. 8, 11, and 13,with the modification being the arrangement of the plug-ins on thesynchronous thread (from each being daisy-chained to each being calledseparately by the real time stylus object).

-   -   a. The real time stylus object 603 checks for plug-in data        objects first on its input queue 604 and then from the tablet        pen data stream B.    -   b. The real time stylus object 603 sends C one plug-in data        object to the objects 606-608 in its synchronous plug-in        collection. Each synchronous 606-608 plug-in can add data to        either the input queue 604 or output queue 605.    -   c. Once the plug-in data object has been sent to all members of        the synchronous plug-in collection, the plug-in data object is        placed on the real time stylus object's output queue 605.    -   d. The real time stylus object 603 then checks for the next        plug-in data object to process (from input queue 604 or data        stream B).    -   e. While the real time stylus object's output queue 605 contains        data, the real time stylus object 603 sends one plug-in data        object from its output queue 605 to the objects in its        asynchronous plug-in collection 610-611. Each asynchronous        plug-in 610-611 can add data to either the input queue 604 or        output queue 605. But since the asynchronous plug-ins may run on        the UI thread, the data added to the queues 604/605 do not have        a set relationship to the rest of the data in the tablet pen        data stream B or to the real time stylus object's input 604 and        output queues 605.

Fourth, the system includes an asynchronous plug-in 611 (representedhere as an ink collection object). The ink collection object here may berepresentative of one or more plug-in objects. Ink collection andstorage may be one of various actions that occur in the UI orasynchronous thread. Once the packet data (or modified packet datareturns from the synchronous plug-ins 606-608, it is placed in theoutput queue 605 of the real time stylus 603. The real time stylus 603then places the data in the output queue 605. The data is then forwardedto the next set of plug-ins 610-611 (in a collection or chain). This mayinclude forwarding the data to the ink collecting object 611 where itmay be destroyed/deleted/recycled/freed as specified by the variousplug-ins on the asynchronous thread.

Fifth, the system may include standard plug-ins (which may be designedto run on either or both of the synchronous or asynchronous threads)that provide basic functionality to the system. The standard plug-insmay include a dynamic renderer 608 and a gesture recognizer 610. Ofcourse, the standard plug-ins may be replaced with other plug-ins ifdesired. Further, in some implementations of aspects of the invention,no standard plug-ins may be initially included with the real timestylus. Rather, developers may be encouraged to select from third partysuppliers and/or create their own plug-ins as needed.

FIG. 7 and 8 provide an alternative version of the structure of FIG. 6.FIG. 7 shows pen services 701 providing stylus data to real time stylus702. Input queue 703 (which may or may not be used) provides the stylusdata to synchronous plug-ins 801-805. Here, each plug-in is part of aplug-in collection. Alternatively, the plug-ins may be part of a chainof plug-ins. After the last synchronous plug-in (real time stylus syncplug-in N 805), the stylus data (or modified stylus data) is placed inoutput queue 709 then handled by the subsequent real time stylus eventasynchronous plug-ins 710 and 808.

With respect to the data of FIGS. 6 and 7, each data set as transmittedthrough the RTS may be a single data set or a bundle of data sets thatare bundled together for efficiency since the digitizer is sampling at avery high rate. These data sets provide notifications of various eventsor forward new information through the RTS. In some situations, the dataset may be deleted. In other situations, where the data set is a bundleof data sets, a single data set in the bundle may be deleted whileretaining the other data sets. Effectively, the bundle of data sets ismodified. The real time stylus 702 then may post a private message tothe window to which it is attached and proceed to the next data set inthe input queue (or if none, returns from the function called by penservices 701 on the pen client interface.

FIGS. 8A and 8B show data flows for various operations. FIG. 8A shows acolor change being received in pen services 701. Real time stylus 702includes a synchronous plug-in collection 809 and an asynchronousplug-in collection 810. the synchronous plug-in collection includessynchronous plug-in 1 801, dynamic renderer 804 and synchronous plug-in3 805. the output from synchronous plug-in collection 809 is sent tooutput queue 709. Asynchronous plug-in collection 810 receives data setsfrom output queue 709 and processes them in asynchronous plug-ins 1-3811, 812 and 813.

For the following description of FIGS. 8A and 8B, data packets are used.However, it is appreciated that other data sets may be used as well toconvey information. Data packets are but one example of the type of datasets that may be used. Here, the real time stylus 702 has two datapackets A and B in output queue 709. Data packet C is currently beingprocessed by dynamic renderer 804 when a color change CC is received bypen services 701. The color change CC may be immediately forwarded toand processed by one of the asynchronous plug-ins. However, doing so maycreate a confusing situation where data packets A, B, and C were createdprior to the color change. Accordingly, one may desire to process thecolor change only after the final processing and final rendering of datapackets A, B, and C by asynchronous plug-in collection 810.

To delay the processing of color change CC, one of the synchronousplug-ins in plug-in collection 809 may create and push a data object CC₁into input queue 703. Next that data object may be processed by theinput queue in online with the result in color change being representedby CC₂ in output queue 709. Using this approach, the instruction tochange a color of strokes may be properly ordered with the previouslyreceived data packets.

FIG. 8A shows an example of a color change being handled by the realtime stylus component 702. However, this approach of creating objectsfor handling by an input queue or an output queue is helpful to processother events.

Some plug-ins (including but not limited to the dynamic renderer) mayhave two categories of properties, namely those where changes takeimmediate effect, versus those that take effect from the next stylusdown (stroke start). The separation between the two categories may occurbecause of a variety of reasons including: technical considerations,concern for user experience and user expectations, and the like. Forinstance, a stroke color change may be delayed so as to preserve theorder in which the strokes and events occurred from the user'sperspective.

FIG. 8B shows real time stylus component 702 processing data packets andhandling gesture recognition objects at the same time. Output queue 709includes a number of data objects, including a stylus down event (SD),data packets (P), and a stylus up event (SU). When a stylus up event SUis received from pen services 701 in synchronous plug-in collection 809,gesture recognizer attempts to recognize a gesture from one or moreprevious data packets P. If a data packet has been recognized as agesture, gesture recognizer 814 generates a gesture recognition objectGR and places it in input queue 703. The gesture recognition of objectGR is then sent through the synchronous plug-ins and routed to theoutput queue 709. From the output queue 709, the gesture recognitionobject GR is passed through the asynchronous plug-ins thendestroyed/deleted/recycled/freed. A benefit of the gesture recognitionobject being created and passed back through the input plug-incollection 809 is that the processing allows the received gesturerecognition object to be modified and/or deleted prior to beingforwarded to output queue 709. Further, by having gesture recognitionobject GR handled by the synchronous plug-in collection 809 andasynchronous plug-in collection 810, the data packets between the stylusdown and the stylus up events may be removed because they are supersededby the existence of the gesture object GR. Alternately, the events maybe ignored of the data to which the gesture corresponds might still berelevant to the other plug-ins downstream. In general, the systemmanages which plug-ins are present when the gesture recognizer ispresent so that the behavior is consistent with the desired result ofthe developers, namely that the data that is being ignored is actuallyremoved from a data structure after it has been placed there.

For instance, an InkCollectingObject plugin (which may build up a tableof packets and then creates an ink stroke on SU) may have alreadycreated the stroke by the time the GR gets to it. So, upon receipt ofthe GR, the inkCollectingObject may simply delete the previously createdstroke. Another plugin may then be responsible for triggering theappropriate action on the GR.

The processing of data within the synchronous and asynchronous plug-incollections allows for various handling techniques for data. If somedata was relevant only to some plug-ins, other plug-ins may ignore orpass the data as needed. For example, a first plug-in may determinedthat data packets relate to a non-inkable the region of a display. Asecond plug-in may handle the data packets in a variety of waysincluding:

-   -   a. ignoring the notification from the first that packet,    -   b. absorb the notification and nullify at such that the data        packet is not passed on to subsequent plug-ins, or    -   c. specify that it is not interested in the notification (for        example by setting a property that indicates the extent of        interest the second plug-in has in these types of packets). Here        the real time stylus component may skip the second plug-in as it        has specified that it has no interest in the data packet.

This third approach C offers a performance benefit by removing theoverhead of a function call when none is needed.

Custom stylus data can be added to the real time stylus object bycalling the AddCustomStylusDataToQueue method. If the call to theAddCustomStylusDataToQueue method is made from a synchronous plug-in inresponse to a call to one of its IStylusSyncPlugin methods, then thecustom stylus data is added to the tablet pen data stream in adeterministic manner; otherwise, it is added in an indeterministicmanner. The AddCustomStylusDataToQueue method throws an exception if theRealTimeStylus object is disabled.

Custom stylus data may be added to the real time stylus object's queuesin one of three places.

-   -   a. When the queue parameter is set to “input”, the custom data        is added to the real time stylus object's input queue and is        sent to the synchronous plug-in collection before new data from        the tablet pen data stream.    -   b. When the queue parameter is set to “output”, the custom data        is added to the real time stylus object's output queue after the        data currently being processed by the synchronous plug-in        collection.    -   c. When the queue parameter is set to “output immediate”, the        custom data is added to the real time stylus object's output        queue before the data currently being processed by the        synchronous plug-in collection.

In each of the above cases, data added by subsequent plug-ins in thesynchronous plug-in collection is added after data added by precedingplug-ins.

Custom stylus data is added to the queue as a custom stylus data objectand plug-ins receive this data through theirIStylusSyncPlugin.CustomStylusDataAdded orIStylusAsyncPlugin.CustomStylusDataAdded method.

The dynamic renderer and the gesture recognizer objects may add customstylus data to the queue.

The real time stylus object calls theIStylusSyncPlugin.CustomStylusDataAdded method on the thread from whichit receives the call to its AddCustomStylusDataToQueue method.

The real time stylus object may or may not be configured to collect ink.In the event that it does not collect ink, one can use the real timestylus to forward ink data to an ink collection object. The inkcollection object may be in the form of an ink collection plug-in whichplugs into the output of the real time stylus.

Ink analysis or handwriting recognition is not a function of the realtime stylus object. As the ink-collection plug-in collects and createsink, or as one wants to recognize the ink, one can copy the ink to aRecognizerContext or Divider object.

As described above, ink may be displayed more than once. Ink may bedisplayed the first time with the dynamic renderer. Ink may be displayedthe second time with a static renderer after it has been collected in anink collection object. This may permit other types of renderers to beused to render the ink as it is received from the ink collection object.For instance, one may have multicolored ink as part of the staticrenderer. Pushing this set of effects to the dynamic renderer may be toointensive for the synchronous thread. Accordingly, one may create arenderer and attach it to the asynchronous thread. Alternatively, onemay ignore the default dynamic renderer and create one's own dynamicrenderer and plug it into the synchronous thread. One may also create adynamic renderer which internally reuses a standard dynamic rendererthrough polymorphism, aggregation, or containment as used inobject-oriented approaches.

One may also reuse this new renderer on the asynchronous thread as thestatic renderer or may create a new renderer for this purpose. Forinstance, one may want to render ink as if drawn with charcoal or othertip. Further, one may create a renderer that displays ink as if it wasconstantly changing color over time (quickly or slowly) to represent howphysical ink dries. Moreover, one may create a renderer that displaysink with cycling colors (as in a rainbow) to highlight the ink.Specifically, one may create a dynamic-renderer-plug-in by creating asynchronous plug-in that subscribes to the StylusDown, Packets, andStylusUp notifications. The plug-in may then render the stroke as it isbeing drawn. The new renderer may handle inking as well as variousselection mechanisms.

A dynamic-render plug-in is an object that displays the tablet pen datain real-time as it is being handled by the real time stylus object.Later, for events such as a form refresh, the dynamic renderer plug-inor an ink-collection plug-in may redraw the ink.

The dynamic renderer object implements the IStylusSyncPlugin interface.The dynamic renderer may additionally implement an asynchronousinterface. The dynamic renderer object renders the ink in real-time, asit is being drawn. By default, when the refresh method is called, thedynamic renderer object redraws the stroke currently being collectedalong with any other previously-collected strokes (which wouldcorrespond to DynamicRenderer's CachedData). Other overloads of therefresh behavior are possible that include parameters to constrain thebehavior of a drawing, for instance, with a clipping region so that inkis not rendered outside a given shape.

This is shown in the various figures (that the dynamic renderer objectcan temporarily cache ink data). When the dynamic renderer objectreceives a call to its IStylusSyncPlugin.StylusUp method, it caches thestroke data and adds custom stylus data to the Input queue in responseto the StylusUpData object for the stroke. The CustomStylusData object'sCustomDataId property is set to the DynamicRendererCachedDataGuid value,and the CustomStylusData object's Data property contains aDynamicRendererCachedData object. One may then clear the dynamicrenderer's cache of the associated data once the ink has been rendereddownstream. Alternatively, refreshing the ink (redrawing current strokesand data stored in CachedData) may not always clear the dynamicrenderer's cache of ink strokes because those ink strokes may not yethave passed through and stored by the ink collecting object.

FIGS. 9 and 10 show processes for establishing aspects of the presentsystem. In FIG. 9, step 901, a real time stylus component isinstantiated. In step 902, synchronous and asynchronous plug-ins areinstantiated. This may or may not include the instantiation of a dynamicrenderer and/or gesture recognizer 903. In step 904, plug-ins are addedto plug-in collections. The real time stylus is enabled in step 905. Instep 906, pen data is received through the real time stylus.

FIG. 10 shows a process for using the real time stylus. In step 1001, apacket is received by the RTS. This may or may not include the operationof the input queue. The input queue supports insertion of arbitrary dataat any time such that it will go through the synchronous collection. Inthis example, the only component putting data into the input queue ispen services, as a result of, for instance, calling a function on RTS'simplementation of a pen client interface.

In step 1002, the packet enters the synchronous plug-in collection. Forsystems such as that of FIG. 8, this step may mean forwarding the packetto the first of the plug-ins. The real time stylus calls the functionappropriate for the particular packet data on the first plug-in in thecollection (or chain if a chain is present). In many cases, thiscomponent has the option of modifying the data or leaving it alone. Asthe component completes its operation, it calls the same function on thenext plug-in object, and so on.

In step 1003, the packet proceeds through the synchronous plug-incollection (or chain). This may or may not include packets forwarded toa gesture recognizer or dynamic renderer as shown in step 1004. If adynamic renderer is used, it may start accumulating packet data on acursor down event (also referred to as a pen down event) in a storageand beings rendering them on a screen (the operation of the dynamicrenderer). On a cursor up event (or pen up event), the dynamic renderermay clear its storage for the next cursor down event.

If a gesture recognizer is used, the gesture recognizer may listen for acursor down event and begin accumulating data in its own storage (forinstance, storage 807). As strokes accumulate, the gesture recognizerpasses the strokes into a recognizer and, if the strokes correspond to adesired pattern (or gesture), then a) the gesture data is added to theoutput queue and b) the storage is emptied and accumulation begins againwith the next cursor down event passes through the queue. Here, it isnoted that gestures may include one or more strokes. FIG. 14, describedin greater detail below, shows how multiple stroke gestures may behandled with respect to the process of FIG. 10.

To what extent a plug-in should or can be aware of previous strokesdepends on the purpose of the plug-in.

-   -   a. In one example scenario, a plug-in that is listening for a        gesture might be responsible for performing an action in        response to the gesture. It might not need to know anything        about the strokes. For instance, if a “circle” gesture in a        window means “delete” for the application, then the plug-in that        receives the gesture would simply send a “delete” keystroke to        the window. For this example scenario it may be that ink is not        even being collected in the window; it might be a window        containing text and the RTS framework may be arranged in this        case to allow stroke data to simply fall through, never being        collected. Thus the plug-in has no need to cache the stroke.        -   i. In an alternative case the “interpreter” plug-in above            might generate its own notification so that downstream            plug-ins would know that the gesture was interpreted and            thus not persist its stroke(s).    -   b. In another example scenario, the plug-in might be performing        ink collection and editing operations. A canonical example of        this might be a mode (an ink and gesture mode) of the ink        collector object where all strokes are potentially gestures,        and, if they are treated as gestures, then the stroke is        subsequently deleted. Thus the plug-in should cache strokes in        order to be able to find and delete them should those strokes        end up being interpreted as a gesture.

In both of these example scenarios, the distinction between single andmultiple stroke gestures does not really matter; the applicationscenario may call for multiple-stroke gestures (e.g. arrow) in whichcase both of the example plug-ins above will be aware of multiple-strokegestures out of necessity.

Next, in step 1005, the packet enters the output queue after beingprocessed by the collection of synchronous plug-ins. As describedpreviously, the collection of plug-ins may, in handling each of theobjects sent to it, modify delete and/or replace the data packet.Assuming that none of the synchronous plug-ins deleted the packet, step1005 is executed.

In step 1006, the packet proceeds through the asynchronous plug-in incollection.

Dynamic Rendering And Wet Ink

The system provides the ability to handle stylus data quickly andefficiently. To make electronic ink behave like real ink, the electronicink needs to appear to flow as easily as real ink from the tip of astylus. Small disruptions or delays in the flow of ink are detrimentalto the electronic inking experience. The follow describes variousapproaches to ensuring electronic ink appears to flow smoothly from thestylus. “Wet ink” is considered the ability to control how long thedynamic renderer owns static rendering after collecting ink. To helphandle the management the lifetime of ink in the dynamic renderer, anotification may be used (referred to herein asDynamicRendererCachedData) allows an ink collecting plug-in to free thecached (or wet) ink from the dynamic renderer.

A variety of different types of dynamic renderers may be created. Forinstance, instead of rendering ink as having been created by a finepoint or chisel point pen, a dynamic renderer may render ink as havingbeen created by translucent, water-based paint (also referred to as“water colors”). Additionally, a dynamic renderer may be created thatrenders ink as having been created by a charcoal stylus. Further, thedynamic renderer may render ink such that it is not fixed, but regularlychanges colors and/or moves. For instance, the dynamic renderer mayrender ink appearing as small drops of ink or dust that coalesce intothe desired ink form, as small worms that eventually move together toform desired ink, or as ink that changes color and shape.

The dynamic renderer may display ink in the same form as that later tobe rendered by an object associated with an asynchronous thread of theRTS. Alternatively, a relationship may exist between the ink rendered bythe dynamic renderer and final display of ink associated with theasynchronous thread of the RTS. For instance, the dynamic renderer mayshow ink in a first shade or shape, only to be later modified by theplug-in on the asynchronous thread to a second shade or shape. This mayappear to the user as drying of ink or paint as flowing from a stylus.

The following description of dynamic rendering and wet ink may beapplied to FIGS. 6-7. For purposes of explanation, FIG. 6 is referenced.The arrangement of FIG. 7 is equally applicable and the followingdescription is desired to encompass FIG. 7 as well.

Referring to FIG. 6, digitizer input enters from pen services 601 andreaches real time stylus 603. The real time stylus passes the digitizerinput through multiple synchronous plug-in objects 606-608 and storesthe result into output queue 605. Objects 606-608 may be connectedthrough an interface that describes the link between the objects606-608, for instance, IStylusSyncPlugin. The asynchronous plug-incollection of objects 610-611 then receives the output from output queue605 and begins to process it. For instance, ink collecting object 611many extract digitizer input from the output queue and store it as ink.Dynamic rendering of the digitizer input occurs when real time styluscomponent 603 passes the digitizer input (as modified or not) through todynamic renderer 608. Static rendering of the digitizer input (asmodified or not) occurs when the digitizer input is stored in the inkstorage inside of the ink collecting object 611.

One issue that may occur is where an inking surface may becomeinvalidated after dynamic rendering 608 and prior to the digitizer inputbeing stored inside of the ink collecting object 611. The resulting userexperience would be displayed ink vanishing until the digitizer inputreaches the ink collecting object 611. The user may believe he or sheneeds to re-create the ink and then spend time doing so, resulting induplicate ink eventually being stored. This visual hiccup is due to theink not being readily shown to a user or shown after a delay.

In order to avoid this vanishing ink scenario, a communication protocolmay be established between the dynamic renderer 608 and ink collectingobject 611. The dynamic renderer 608 may continue to cache digitizerinput in visual store 609 until it is told that it can release thedigitizer input by the ink collecting object 611 (for instance, when thedigitizer input has been stored in ink store 612).

For purposes herein, the temporal cache 609 of digitizer input in thedynamic renderer 608 is currently referred to as cacheddata. Wet ink isthe ability for the system to render ink prior to digitizer input beingstored in the ink collection object 611.

FIG. 11 shows how the ink collection object alerts the dynamic rendererto release its cached data. Dynamic renderer 1201 includes visual store1202. Ink collecting object includes ink store 1204. Once the dynamicrenderer 1201 has stored stylus data in visual store 1202, it outputs anobject into the input queue 1205. The real time stylus (for instance,702, probes the input queue for the next data to process prior tolooking to the pen services 701. For purposes herein, the objectinjected into the input queue is referred to asDynamicRendererCachedData. The object is then picked up by the real timestylus 702 (with or without additional processing) and outputs it intooutput queue 1206. Meanwhile, the ink collecting object 1203 isprocessing the received stylus input packets in order of appearing inthe output queue 1206. When the ink collecting object 1203 encountersthe object from the dynamic renderer 1201 (here, theDynamicRendererCachedData object), it performs the following:

-   -   a. creates a proper correlating stroke that matches the stroke        rendered by the dynamic renderer;    -   b. applies the drawing attributes present on the dynamic        renderer stroke to the stroke to be rendered; and    -   c. sends a release notification to the dynamic renderer 1201.        This notification may involve executing a method on the Dynamic        Renderer and passing a parameter.

Step B above may or may not perform properly based on whether or not thesynchronization of drawing attribute modifications have occurred priorto all strokes having been moved out of an output queue. FIG. 8A relatesto a process for synchronizing drawing attribute changes with previouslyreceived data packets.

The drawing attributes may be set in the system in accordance with adeveloper's desires or, among other approaches, may be provided in theobject passed to the ink collecting object 1203. The releasenotification may be a call on an interface provided by the dynamicrenderer 1201. Here, the notification may be ReleaseCachedData.

FIG. 12 shows an alternative approach to handling wet ink. FIG. 12includes the pen services input 701 sending stylus data to real timestylus 702. Input queue 703 may handle and queue inputs with results ofreal time stylus event sinks being placed in output queue 709. FIG. 12includes an additional queue 1101. This queue handles the results fromdynamic renderer 1102 which has been separated from other real timestylus event sinks A 1103 and B 1104. Here, this approach addresses theissue that may occur in which real time stylus event sinks A 1103 and/orB 1104 slow down the appearance of ink flowing from the stylus. Thesystem of FIG. 12 handles the dynamic rendering of stylus data before ithandles other event sinks A 1103 and B 1104. Even though event sinks A1103 and B 1104 may still cause delays in the processing of stylus dataon its way to the ink collection object 1105, the dynamic rendering 1102event sink may continue to process stylus data as it is received. Here,as soon as stylus data is rendered by dynamic renderer 1102, the outputis sent to additional queue 1101 where it then is accessed by real timeevent sinks A 1103 and B 1104. Note, the process of how real time eventsinks A 1103 and B 1104 are accessed (one calling the other or bothbeing called by additional queue 1101) can be performed through eitherapproach.

FIG. 13 shows yet another aspect on how to ensure that wet ink flowsfrom a stylus. Here, a secondary real time stylus object exists in aseparate thread. Pen services 701 outputs stylus data to real timestylus component 1301. Input queue 1302 provides stylus data to dynamicrenderer 1303. Other synchronous plug-ins may or may not be associatedwith real time stylus component 1301. Optional synchronous plug-insinclude gesture recognizer 1304 and other synchronous plug-ins 1305. Theoutput from these synchronous plug-ins is forwarded to output queue1306. Here, real time stylus component 1307 is connected to output queue1306 from real time stylus component 1301. The real time styluscomponent 1307 and its associated synchronous plug-ins 1309-1313 andasynchronous plug-ins 1312-1313 acts as an asynchronous plug-in from theperspective of real time stylus component 1301. The data packets(however modified by the synchronous plug-ins of real time styluscomponent 1301) are forwarded to the synchronous plug-in collection forreal time stylus component 1307. For instance, the synchronous plug-incollection for real time stylus component 1307 includes synchronousplug-in A 1309, synchronous plug-in component B, 1310, and (if not usedpreviously) gesture recognizer 1313. It is appreciated that gesturerecognizer 1313 is a type of plug-in that may be associated with asynchronous thread or an asynchronous thread for any real time styluscomponent. Associating the gesture recognizer 1313 with the synchronousplug-in collection for real time stylus component 1307 is forillustrative purposes only.

The output of the synchronous plug-in collection of real time styluscomponent 1307 is forwarded to output queue 1311. Asynchronous plug-inobjects 1312-1313 may then handle the data packets in output queue 1313.Again, ink is dynamically rendered and flows smoothly from the pen evenwhen the application is blocked.

It is noted for reference that the various plug-in components may bechained together instead of following one another in a plug-incollection. An example is shown here in FIG. 13 where synchronousplug-in component A 1309 is chained to synchronous plug-in component B1310. Of course, the approaches of chaining plug-in components togetheror collecting them in a collection may be used in alternatively or inaddition to each other for addressing plug-in components in accordancewith aspects of the present invention

Gesture Recognition

Gesture recognition attempts to determine if a gesture has been made andhandles it appropriately. Gesture recognizer 610, for example, isresponsible for looking at digitizer input, and injecting gesturerecognition results into the input queue similar to the dynamic renderer608. FIG. 8B described how gesture recognition may be performed withsynchronous and asynchronous plug-in collections. That description isrelevant with respect to the operation of the gesture recognitionobject.

The gesture recognizer 610 may use an algorithm. Here, the gesturerecognizer may use a permutation algorithm to recognize multiple-strokegestures. For instance, when a property relating to the maximum numberof strokes has been set, the gesture recognizer looks back that manystrokes. For instance, if the property is set to 2, the gesturerecognizer looks back to the most recent 2 strokes trying to recognizethem as gestures. This may result in multiple recognition calls.However, this approach eliminates a constant delay for having to waitfor a gesture recognizer to start processing after the last stroke.

FIG. 14 shows an example of how far back a gesture recognizer looks backto obtain parts of a gesture. The results may be stored in an object(here, for instance named GestureRecognitionData). The results may beinjected into the real time stylus input queue.

Once the ink collecting object 611 receives the gesture object (here,GestureRecognitionData), it removes the strokes for the gesture from theink store 612 and conducts the appropriate actions in response to thegesture.

As described above, to perform gesture recognition, the system may add aSystemGestureData object to the Input queue in response to the data thatfinishes the gesture, such as a StylusUpData object for the Tap gesture.

The gesture recognizer may be implemented as an object with variousinterfaces. For example, the gesture recognizer object may implementIStylusSyncPlugin and IStylusAsyncPlugin interfaces.

When the gesture recognizer object recognizes a gesture, it adds customstylus data to the Input queue in response to the StylusUpData objectfor the stroke. The CustomStylusData object's CustomDataId property isset to the GestureRecognitionDataGuid value, and the CustomStylusDataobject's Data property contains a GestureRecognitionData object.

By default, the gesture recognizer object only recognizes single-strokegestures. The gesture recognizer object can be set to recognizemultistroke gestures (see FIG. 14, for instance). For multistrokegestures, the CustomStylusData object is added to the Input queue inresponse to the StylusUpData object for the final stroke of the gesture.When recognizing multistroke gestures, one may receive notifications foroverlapping sets of strokes. For example, the first and second strokestogether may be recognized as one gesture, the second stroke by itselfmay be recognized as a gesture, and the second and third strokestogether may be recognized as another gesture.

If one is using the gesture recognizer object for multi-stroke gesturerecognition, one may use a cascaded real time stylus model and attachthe gesture recognizer object to the secondary real time stylus objectin order to reduce delays on the real-time thread but prevent gesturerecognition from being affected by delays on the user interface thread.

Further, one may create a custom gesture recognizer plug-in thatrecognizes handwriting, gestures, or other objects in three ways.

-   -   a. By passing the stroke information to an existing Recognizer        object and using the AddCustomStylusDataToQueue method to add        the results to the pen data stream.    -   b. By performing the recognition within a custom plug-in and        using the AddCustomStylusDataToQueue method to add the results        to the pen data stream.    -   c. By wrapping a standard gesture recognizer plug-in with a        custom plug-in that calls the standard plug-in in a        daisy-chained fashion. In this way a developer may implement        in-air gesture recognition, by “translating” in-air-packets into        Packets, Cursor-In-Range into StylusDown, and CursorOutOfRange        into StylusUp.

The real time stylus object receives data about system gestures as theyare recognized by the system. The following table describes where theSystemGestureData objects occur in the pen data stream in relation toother pen data. The following list is illustrative and not exhaustive.Other gestures may be used in conjunction with aspects of the presentinvention without departing from the scope of the invention.

SystemGesture Description

Tap After the StylusDownData object, and before the StylusUpData object.DoubleTap After the StylusDownData object, the SystemGestureData objectfor the Tap system gesture, and the StylusUpData object, and before thesecond StylusDownData object. RightTap After the StylusDownData objectand the SystemGestureData object for the HoldEnter system gesture, andbefore the StylusUpData object. Drag After the StylusDownData object,and before the StylusUpData object. RightDrag After the StylusDownDataobject, and before the StylusUpData object. HoldEnter After theStylusDownData object, and before the StylusUpData object. This systemgesture isn't recognized if the user begins a Drag or RightDrag systemgesture. HoldLeave Optional HoverEnter After several InAirPacketDataobjects of low average velocity. There may be noticeable delay beforereceiving the HoverEnter system gesture. The real time stylus objectonly receives this data if the real time stylus object is attached tothe window or control that is directly under the pen at the time of thesystem gesture. HoverLeave After the SystemGestureData object for theHoverEnter system gesture and several InAirPacketsData objects ofsufficient average velocity. There may be noticeable delay beforereceiving the HoverLeave system gesture. The real time stylus objectonly receives this data if the real time stylus object is attached tothe window or control that is directly under the pen at the time of thesystem gesture.Synchronous And Asynchronous Processes

When a real time stylus instance is instantiated, an execution threadmay be instantiated. The execution thread may be separated intosynchronous objects and asynchronous objects. The synchronous objectsgenerally operate in synchronization with packets originating from a penservices component. The asynchronous objects are generally grouped asobjects that do not have to be always executed in syncretism with inkstrokes and or other data packets originating from pen servicecomponents. The separation of these two threads permits a dynamicrenderer object to-quickly handle data packets in synchronization withthe receipt of the data packets and permits other plug-in objects which,while still important, may be handled correctly even with a slight delayfrom the or receipt of the original data packets.

The collection of synchronous plug-ins are executed on this new threadthat was instantiated for the real time stylus instance. The collectionof asynchronous plug-ins may be executed in a user interface thread.

The real time stylus thread may be intended for computationally lightactivities so that inking (packet queuing and dynamic rendering) isgenerally responsive and smooth. Therefore, only computationally lightprocessing should generally be implemented with the thread with for thesynchronous plug-ins.

The user interface thread (usually associated with the output of theoutput queue) is usually used by asynchronous plug-ins that requirecomputationally intensive activities. To assist, a queue architecture isprovided between the real time stylus thread and the user interfacethread to be robust against temporary blockages in the user interface(UI) thread. The UI thread may or may not be the final destination ofpackets and other data collected on the real time stylus thread. It isnoted that alternative architectures may handle computationally intenseactivities better, thereby not readily forcing a developer to compromisebetween blocking a pen thread and a user interface thread. For instancea cascaded design as described herein allows the separation of threadsto better handle different activities.

The gesture recognizer may be implemented on the synchronous thread ormay be implemented on the asynchronous thread if the gesture recognizeris determined or anticipated to be slow from expected heavy gesturerecognition activities (e.g., a large number of accepted gestures towhich strokes are going to be compared against). The gesture recognizerobject may have both a synchronous plug-in interface and an asynchronousplug-in interface to give a developer the flexibility to utilize theobject from either or both threads. This capability is not limited tothe gesture recognizer. The queues. It is appreciated that all plug-insmay or may not have a synchronous interface, an asynchronous interface,or both in order to allow a developer to place the plug-in inasynchronous collection or in an asynchronous collection or both if theplug-in supports this, depending where the developer anticipates a goodlocation for the plug-in to be.

The real time stylus object provides real-time access to the tablet pendata stream and may run on its own thread. Synchronous plug-ins may ormay not generally run on the real time stylus object's thread, whileasynchronous plug-ins may or may not generally run on the application'suser interface (UI) thread. One may separate the synchronous thread fromthe asynchronous thread by placing plug-ins for tasks that requirereal-time access to the data stream and are computationally light, suchas dynamic rendering, on the synchronous thread. Plug-ins for tasks thatdo not require real-time access to the data stream, such as inkcollection, may be placed on the asynchronous thread.

Certain tasks may be computationally intensive yet still require closeto real-time access to the tablet pen data stream, such as multi-strokegesture recognition. Plug-ins to handle these tasks may be grouped onone thread or the other or may be part of cascaded real time stylusobjects as shown in FIGS. 12 and 13.

Cascading

Real time stylus instances are, in many ways, a thread host thatencapsulates extensibility and policy. FIG. 13 shows an example of howmultiple synchronous threads may work together. Gesture recognizer maybe placed on the synchronous thread at location 1304 or may be placed onthe synchronous thread at location 1313.

If there were multiple functionalities that the developer wants toconnect in a multi-threaded manner, the developer can cascade multiplereal time stylus instances and plug their functionality in assynchronous plug-ins.

The specific scenario that suggests this feature is to realize ‘trulyuninterruptible’ inking. The gesture recognizer may at times causesignificant latency to obtain recognition results when the developer isinterested especially in multi-stroke gestures. Therefore, the gesturerecognizer may not be on the synchronous collection of real time styluswhere the dynamic renderer is positioned, as it has a potential to blockdynamic rendering. The gesture recognizer may be also located with theink collection object with the collection of asynchronous plug-ins.

The system as shown in FIG. 13 is beneficial where one wants to separatecomputationally intensive operations from the gesture recognizer orother threads for operations including, such as, accessing a database,posting to web server, refreshing a screen, and the like. Where gesturerecognition is an integral part of the ‘truly uninterruptible’experience, one may have the gesture recognizer run on its own thread,namely a second real time stylus thread.

Real time stylus may support this usage by implementing an asynchronousinterface, thus allowing it to behave as a mere asynchronous plug-ininstance. The following describes how a real time stylus may be added toanother real time stylus. The following provides a number of optionsthat may or may not be implemented.

-   -   1. When a real time stylus is added to the collection of        asynchronous plug-ins, the system may limit relationships such        that no other asynchronous plug-in interfaces may be added.    -   2. Different numbers of real time stylus objects may be cascaded        along multiple dimensions. For instance, multiple RealTimeStylus        objects could be connected directly to a single “parent”        RealTimeStylus, or alternatively the multiple RealTimeStyluses        could be connected in series such that each cascades into the        next, or alternatively the RealTimeStylus objects could be        connected in a combination of the aforementioned configurations.        An implementor could choose to constrain these combinations for        simplicity or reliability.    -   3. When the same real time stylus instance is added to multiple        collections of asynchronous plug-ins from multiple real time        stylus parents, the cascading real time stylus may function as        normal. Alternatively, the cascading (child) real time stylus        instance may stop functioning and insert an ErrorData object        into the queue. The ErrorData object may be an indication to        downstream objects that there is an error in the data or that a        previous plug-in encountered an error while processing the data.        Of course, this is but one approach to handling errors. Other        approaches may be used as well.

It is noted that, while one implementation may re-use an asynchronousplug-in collection as a collection point for cascading, an alternateimplementation may have a dedicated connection point for connecting acascaded RTS, thereby allowing each RTS in the cascade to have its ownindependent set of synchronous and asynchronous plug-ins.

It is appreciated that the existence of the ErrorData object may beindependent from cascading. So, the generation of the ErrorData objectmay occur with respect to a single real time stylus component as well aswith cascaded real time stylus components.

Dynamic Plug-In Collection Modification

The synchronous and asynchronous plug-in collections on the real timestylus may be modified without disabling and then re-enabling the realtime stylus for performance benefits. Thus, there is no assurance thatthere is an ‘enabled’ or ‘disabled’ operation done at the real timestylus scope.

In order to ensure a timing where plug-ins can initialize & clean-up,‘artificial’ (meaning not resulting from true client code calls toenable & disable) RealTimeStylusEnabled & RealTimeStylusDisabled callsmay be made to plug-ins which are dynamically inserted or removed fromthe plug-in collections.

The following describes some of the behaviors that may occur when addingor removing plug-ins.

-   -   When a plug-in is added to an enabled real time stylus instance,        an artificial RealTimeStylusEnabled call may be made to this        plug-in;    -   When a plug-in is added to a disabled real time stylus instance,        nothing may happen;    -   When a plug-in is removed from an enabled real time stylus        instance, an artificial RealTimeStylusDisabled call may be made        to this plug-in; and    -   When a plug-in is removed from a disabled real time stylus        instance, nothing may happen.

The way these relationships are maintained may or may not includeincrementing a counter each time it gets a real time stylus enabledcall. Every time it gets a real time stylus disabled call, this counteris decremented. When the counter is 1, this plug-in functions. When thecounter becomes larger than 1, this plug-in raises an exception andinhibits its functionality. When the counter comes back down to 1, thisplug-in stays disabled. The plug-in only re-enables itself when thecounter is back down to 0 again. Other approaches may be used.

Error Propagation

Design Time

While in a development environment (for instance, Visual Studio.NET byMicrosoft Corporation) developers may break at any Exception that israised regardless of whether it is caught or not with a try catch.Therefore, error reporting is straightforward for the purpose ofdetecting invalid configurations of the RealTimeStylus framework.

Run Time

The error reporting/propagation issue may become difficult for run timeerrors, as real time stylus and synchronous plug-ins are running on thereal time stylus thread (ink thread) which is different from the UIthread (app thread). If a synchronous plug-in were to raise anexception, it can be caught by the real time stylus but the exceptionhas no where to go beyond that since it is on the real time stylusthread (ink thread), and the origin of the control flow is the real timestylus. Propagating the error to the user interface thread and allowingthe client code to gracefully deal with the error may be difficult. Thefollowing describe a solution.

The .NET standard error handling is to fire events from the plug-inswhen exceptions occur, and have the developer's error handling code inthe UI thread listen to this event. This does not work for Real TimeStylus though since when a plug-in fires an event, there is apossibility that the data that caused the exception may not have reachedthe UI thread due to the queuing. It is challenging to conduct gracefulerror handling without the availability of the context (namely the baddata as well as preceding and succeeding data) in which the exceptionoccurred to the error handling code. This approach is only good forsimplistic error handling such as raising an error dialog andterminating the application.

The proper places to conduct error handling within the real time stylusframework is in the plug-ins (synchronous or asynchronous), as theplug-in is the only code blocks that are under the developer's control.

The real time stylus thread (ink thread) should not immediately stopfunctioning on an exception raised from a synchronous plug-in as thismakes it impossible to conduct error handling in the UI thread (appthread) (e.g. the queues need to stay intact for error handling).Furthermore, what one plug-in considers invalid input may be completelyvalid for another plug-in.

FIG. 15 shows a possible solution. Here, exception information isre-routed through the queues as an ErrorData object. Error handling maybe conducted in plug-ins. The exception information may be processedthrough the data flow created for plug-ins, namely the queue.

As shown in FIG. 15, the real time stylus 603 catches all exceptionscoming from plug-ins 606-608 and creates an ErrorData object. Thefollowing shows a process for identifying errors:

-   -   a. If the exception occurred in the error method on the        interface for the synchronous or asynchronous plug-ins, no        ErrorData object is created to avoid infinite loops by calling        back into the exception origin plug-in.        -   i. An alternative approach is to immediately pass the            ErrorData object through the remaining plug-ins in the            collection, prior to the actual data that caused the            exception.    -   b. If the exception occurred in other methods on the interface        for the synchronous or asynchronous plug-ins methods, the        ErrorData object may be immediately passed to the exception        origin plug-in itself first, and then passed through the        remaining plug-ins in the collection, prior to the actual data        that caused the exception.    -   c. The ErrorData is eventually queued in the output queue if the        exception occurred in the synchronous plug-ins, or delivered to        asynchronous plug-ins.    -   d. After the ErrorData is through the collection, the data that        caused the exception passes through the remaining plug-ins in        the collection.

The ErrorData object may be passed through the plug-in collectionthrough dedicated methods (for instance, IStylusSyncPlugin.Error methodor IStylusAsyncPlugin.Error method).

More particularly, when a plug-in throws an exception, the normal flowof data is interrupted. The real time stylus object generates anErrorData object and calls the IStylusSyncPlugin.Error orIStylusAsyncPlugin.Error method of the plug-in that threw the exceptionand the IStylusSyncPlugin.Error or IStylusAsyncPlugin.Error method ofthe remaining plug-ins in that collection. If the plug-in that threw theexception is a synchronous plug-in, the ErrorData object is added to theoutput queue. Then real time stylus object resumes normal processing ofthe original data.

If a plug-in throws an exception from its error method, the real timestylus object catches the exception but does not generate a new errordata object. Alternatively, a new error data object may be produceddepending on the implementation or desire of the developer. In thisalternative approach, the real time stylus may then reduce any createdloop by limiting the number of recursions or it may ignore the riskaltogether.

In relationship to custom stylus data added by synchronous plug-ins, theerror data is added to the output queue after any custom stylus datathat is added to the “output immediate” queue prior to the exceptionthat created the error data and before any custom stylus data that isadded to the “output immediate” queue by subsequent plug-ins in thesynchronous plug-in collection. If any of the synchronous plug-ins addscustom stylus data to the “output immediate” queue in response to theerror data, the data is added immediately before the error data. If anyof the synchronous plug-ins adds custom stylus data to the output queuein response to the error data, the data is added immediately after theerror data.

The real time stylus object calls the IStylusSyncPlugin.Error method onthe thread from which the exception is thrown.

Managed/Unmanaged Illustrations

FIG. 16 shows various approaches to implanting aspects of the presentinvention. The system may include COM objects wrapped by a collection ofC# managed objects. Alternatively, any object orient language may beused including Java, C++, and the like.

FIG. 16 shows a real time stylus component existing in both managed andunmanaged space. FIG. 16 includes a dynamic renderer, asynchronousplug-in Y, an asynchronous plug-in and a gesture recognizer.

Pen services 1601 sends stylus data to real time stylus component 1602with its input queue 1603 and output queue 1604 in the unmanaged coderegion. Stylus data is passed to the real time stylus component 1605 inthe managed code region.

The dynamic renderer is the first synchronous plug-in attached to thereal time stylus component. In the unmanaged space, the dynamic renderer1608 with its data cache 1616 is attached to the real time styluscomponent 1602. Similarly in the managed space dynamic renderer 1615with its data cache 1617 is part of the synchronous plug-in collectionfor real time stylus component 1605. The next synchronous plug-in in thesynchronous plug-in collection is synchronous plug-in 1607. Synchronousplug-in 1607 follows after dynamic renderer in synchronous plug-incollection for the RealTimeStylus component 1605. Because synchronousplug-in Y 1607 only exists in managed space, a synchronous plug-inwrapper 1618 allows the unmanaged RealTimeStylus component 1602 toaccess synchronous plug-in 1607 over the managed/unmanaged boundary.

FIG. 16 also shows a collection of asynchronous plug-ins. Theasynchronous plug-ins include a synchronous plug-in 1612 and gesturerecognizer 1609. Asynchronous plug-in 1612 is the first asynchronousplug-in in the asynchronous plug-in collection attached toRealTimeStylus component 1605 in managed space. Because asynchronousplug-in 1612 is in managed space, asynchronous plug-in wrapper 1619 maybe used to allow access to it from the unmanaged RealTimeStyluscomponent 1602. The gesture recognizer exists both in managed space andunmanaged space. The managed gesture recognizer 1609 with its data cache1610 is the next plug-in accessed after asynchronous plug-in 1612.Gesture recognizer 1609 may exchange information with the unmanagedversion of the gesture recognizer (1613 with its data cache 1614).

Data crossing between the managed and unmanaged sides of FIG. 16 mayneed to be converted or “marshaled” between the structures used inunmanaged space and structures used in managed space. The .NET Frameworkby Microsoft Corporation provides an interoperability layer thatperforms much of this marshaling automatically. This additional dataprocessing incurs an implied performance penalty, so the design as shownin FIG. 16 is tuned to minimize the number of interoperability layercrossings.

The relationship between managed wrapper for RTS 1605 and unmanaged RTS1602 is that to the native (unmanaged RTS 1602), the managed RTS 1605looks like another RTS event sink. When instantiated with dynamicrenderer 1615 in its constructor, the RTS 1605 accesses thecorresponding unmanaged dynamic renderer 1608 and hooks it in tosynchronous plug-in collection after itself

The managed gesture recognizer object 1609 may be a full implementationof an asynchronous plug-in. It accumulates packet data on the managedside and marshals it across the interoperability layer, and passes intothe unmanaged code implementation 1613. The return code is whetherpackets correspond to a gesture or not.

The managed and unmanaged dynamic renderer objects 1615 and 1608 arealso shown in FIG. 16. The managed dynamic renderer object may be a thinwrapper over the properties of the unmanaged dynamic renderer 1608. Themanaged dynamic renderer 1615 is optional. If the dynamic renderer 1615is not instantiated here, it may be instantiated as one of the othersynchronous plug-ins.

The following is a process for creating the system of FIG. 16:

-   -   a. First, a developer instantiates the managed asynchronous        plug-in 1612 and managed dynamic renderer 1615. Internally,        dynamic renderer 1615 instantiates an unmanaged dynamic renderer        1608 in order to pass property settings down.    -   b. Second, a developer sets properties on the dynamic renderer        1615 (drawing attributes, etc).    -   c. Third, a developer instantiates real time stylus 1602,        handing the dynamic renderer 1615 in the real time stylus        constructor. A few things may happen internally:        -   i. Managed real time stylus 1605 queries (via public sealed            accessor) the address of the unmanaged dynamic renderer            1608.        -   ii. Managed real time stylus instantiates a native real time            stylus 1602, which hooks itself to pen services 1601.        -   iii. Managed real time stylus 1605 hooks itself to the            native real time stylus 1602 as a real time stylus event            sink.        -   iv. Managed real time stylus 1605 hooks the native dynamic            renderer 1608 to its synchronous output.        -   v. Managed real time stylus 1605 hooks itself to the native            real time stylus 1602 synchronous plug-in thread.    -   d. Fourth, the developer instantiates one or more managed        synchronous and/or asynchronous plug-in objects, optionally        including the managed gesture recognizer 1609.    -   e. Fifth, the developer collects the managed plug-in objects        together, into various plug-in collections. (This may be done by        informing the RealTimeStylus components of the order in which        the plug-ins should be accessed. Alternatively, for chaining the        plug-ins together, one may set a property and in this reference        to the first plug-in.)    -   f. Sixth, the developer associates the managed asynchronous        plug-in object 1612 with the managed real time stylus 1605        InkCollectionObject property (thus hooking the asynchronous        plug-in object 1612 to the asynchronous plug-in collection of).    -   g. Seventh, the developer sets RTS.Enabled to “true”. This may        also cause the managed real time stylus 1605 to set the        IRealTimeStylus→Enabled to “true” on the native real time stylus        1602.    -   h. Eighth, events begin streaming in through real time stylus        1602.

Step (c) may be realized with an alternative arrangement. For instance,the API may not have any plug-in specific accommodations on the RTSconstructor. Instead, the various overloads of the RTS constructordictate how the RTS itself will be used (e.g. attached to a window orcontrol, associated with one or many digitizers, mouse verses penbehavior, etc. The DynamicRenderer parameter may be used because ofperformance implications of data crossing the interop boundary multipletimes. As a result DynamicRenderer may be used in an architectural senseto be the first or last synchronous plug-in in the plug-in collection orchain.

FIG. 16 shows an approach of an architecture that minimizes the interopcrossings and allows the DynamicRenderer to be anywhere in the orderedlist of plug-ins.

The following describes a process for using the system of FIG. 16:

-   -   a. Pen service 1601 calls a function on RTS 1602, passing data        which is accumulated into the synchronous plug-in collection        associated with RealTimeStylus component 1602. In some cases,        data from input queue 1603 may also be passed into the plug-in        collection.    -   b. Native RTS 1602 does the following when new data appears in        the input queue 1603 (or when the appropriate method is called        on the interface it exposes to pen services 1601):        -   i. transform packets from digitizer space into himetric            (“ink space”);        -   ii. embed the inverse transform (himetric→digitizer) in the            packet data; and        -   iii. pass the transformed packet data into the first            synchronous plug-in of the plug-in collection can the by            calling the appropriate function for the data on the            synchronous plug-in collection interface (Managed RTS in            this case).    -   c. Managed RTS 1605 does the following:        -   i. Creates the appropriate managed data structures to hold            the data in a managed-friendly way;        -   ii. Calls the appropriate function on the collected managed            synchronous plug-ins;        -   iii. Ensures that each plug-in processes the data and            returns back to RealTimeStylus component 1605 so that the            next plug-in may be called; and        -   iv. And so on until it calls the corresponding function on            gesture recognizer 1609 (if gesture recognizer 1609 is being            used).    -   d. Dynamic renderer 1615 may be called by the RealTimeStylus        1605. When dynamic renderer 1615 is finished rendering the new        packets, it returns, the unmanaged RTS 1602 detects whether        packet data was modified, and puts the appropriate data into the        queue. If there were any “non-immediate” user data items to add,        it adds them now.    -   e. Native RTS 1602 then posts the private message to the        attached window; and    -   f. Native RTS 1602 then looks in the input queue 1603 to see if        any real-time data was added and if so calls the synchronous        plug-in collection again passing that data. If no data was        added, native RTS 1602 returns, allowing pen services 1601 to        proceed and call RTS 1602 again with new digitizer data.    -   g. The private message is seen by the subclass proc on native        RTS 1602, which retrieves the data from the queue 1603 and calls        the appropriate function on the attached synchronous plug-in        interface.    -   h. In this case, the managed RTS 1605 has connected itself and        is called. The managed RTS 1605 then calls through to the        corresponding function on the asynchronous plug-in 1612. This        may be passed through asynchronous plug-in wrapper 1619.    -   i. The managed asynchronous plug-in 1612 then processes the data        asynchronously, rendering or accumulating as appropriate.    -   j. The gesture recognizer 1609 may be called as part of the        asynchronous plug-in collection off the RealTimeStylus        component. The gesture recognizer 1609 internally accumulates        packets until a CursorUp is encountered, at which point the        accumulated packets are passed across the interoperability        boundary into the native gesture recognizer 1613 for        recognition.        -   i. This operation returns information about which gesture(s)            were recognized, and how many strokes correspond to each of            the possible gestures;        -   ii. Gesture recognizer 1609 then uses an AddUserDataToQueue            method on managed RTS 1605 (with “immediate” set to false)            to put the results of the gesture recognition into the queue            1604 for asynchronous consumption. (This causes the new            gesture data to cross the interoperability boundary (or            interop) immediately and sit in another queue until the            native plug-in collection is complete); and        -   iii. It additionally fires a traditional managed event to            any listeners to notify the world in real time that a            gesture just happened.        -   iv. The current data is then returned by gesture recognizer            1609 to the RealTimeStylus component 1605 for any additional            plug-ins in the plug-in collection.            Data Sets and Flows

The following lists event sink handlers and information associated withthem:

-   -   Source IRealTimePen This back pointer gives the event sink        implementation a back-reference to the source RealTimeStylus.        This pointer allows two things: (a) access context information        about the stylus instance (for instance, Dynamic Renderer 1615        needs to be able to do unit conversions between digitizer        coordinates and pixels), and (b) be shared among multiple RTS        event chains.    -   TabletContextID Allows the consumer of real-time data to        efficiently accumulate data and use the tablet ID as an indexer        in the data collection    -   PacketDescription Describes the layout of data in a packet call.        Different digitizers support different amounts of information,        including (for instance) x, y, pressure, angle. The        PacketDescription indicates to the receiver of a packet how this        data is arranged in the flat array of integers that eventually        arrives.    -   CursorID Allows the consumer of real-time data to efficiently        accumulate data and use the cursor ID as an indexer for the data        collection.    -   StylusInfo A simple structure that consolidates convenient        information including:        -   TabletContextID;        -   CursorID;        -   Inverted State (this is a tri-state indicating relevance and            whether the cursor corresponds to the “eraser” end or            “writing” end of a pen);        -   Button States (whether each of up to 32 buttons is pressed);            and        -   Other relevant information as needed.    -   PropertyCountPerPacket Compromise parameter that allows a        developer to infer the layout of packet data without an API        call. (X, Y guaranteed to be the first and in that order, and        count allows developer interested only in xy data to walk        through the list skipping every n^(th) element.    -   CountOfPackets The number of packets “bundled together” for        efficiency (Pen services and/or the digitizer device driver        determines when and how many packets are bundled in this way).    -   PacketData This is a read-only copy of the packet data.    -   CountOfPacketsRef The number of packets in the modified        packetdata set which plug-in developers can allocate.    -   PacketDataRef This is a reference to the packet data which gives        developers of plug-in objects the ability to modify the packet        data in order to accomplish scenarios that involve real-time        data manipulation.

In one implementation, one may not allow plug-in collectionimplementations to “know” whether packet data had been modified prior toentering the plug-in collection. Alternatively, the plug-in collectionsmay be informed if the packet data has been modified. For instance, aflag on the data or a flag independent of the data may be set. Oneadvantage of allowing plug-in collections the knowledge of whetherpacket data has been modified provides plug-in collections the abilityto ignore modified packet data out of caution. On the other hand, thisapproach provides further options for a developer to make mistakes. Moreimportantly, it muddles the simplicity and object oriented nature of thedesign pattern because plug-ins are not longer agnostic to the origin ofthe data. If the plug-in can inspect both the original and the modifieddata, then it is probably performing operations that belong in theprevious plug-in.

For performance/efficiency reasons, one may internally maintain two datastructures, so that unless the developer intends to modify the data onedoes not need to allocate the memory.

Data Synchronization

The following describes how data may be added to a queue to ensuresynchronization as shown in FIG. 17A.

An ICO may use the input and output queues as a signaling mechanism, inorder to synchronize other requests that come in on the UI thread withpending data coming through the inking thread.

FIG. 17A shows a mode switch that occurs while the system is processinginformation. The system shows the conclusion of a first processing(“done”) at 1701. At time 1702, new stylus data is received and startsbeing processed by plug-in 1 at time 1702 and plug-in 2 at time 1704 andfinishes at time 1705. Mode switching occurs at time 1703. For instance,if the user of an application is changing the “editing mode” from “ink”to “erase” and the application is busy, there is a reasonable likelihoodthat packet data streaming from the queue is still “ink” for some amountof time after the user mode changed.

However, instead of adding the mode switch event to queue 1706 at thepresent time, it is delayed until after the current processing isfinished at time 1705 for stylus data. This may occur two ways. First,the mode switch operation event may be delayed until the stylus datastarting in the plug-ins at time 1702 is completed. Second, the systemmay input the mode switch event in queue 1706 leaving enough room forthe result from plug-in 2 at time 1705 to be placed in front of it inthe queue 1706. So, in this case, the ICO may insert a “marker” into theoutput queue at the instant that the user made the mode change. After aperiod of time has elapsed that marker will be passed back into the ICO(via the CustomDataAdded method on real time stylus plug-in collectioninterface), at which point the ICO can begin interpreting incomingpacket data as “eraser” requests.

FIG. 17B shows a generic version of FIG. 17A. The end of a process for afirst thread is shown by event 1707. It gets placed in queue 1712. Nextat time 1708, another process on the first thread starts. At time 1709,an event occurs on a second thread. Because the process on the firstthread started at point 1708, one expects that process to be addressedprior to the event at time 1709. However, because the process on thefirst thread continues at time 1710 then ends at time 1711, timing theinsertion of the event at time 1709 to the time of the start of theprocess 1708 would be detrimental in that the event at time 1709 wouldbe placed in the queue prior to the process being done at time 1711.Here, FIG. 17B shows the event at time 1711 being inserted a distancedownstream in queue 1712 from the event generated from process donepoint 1707 by skipping at least one place in queue 1712. By keeping aplace available in queue 1712 allows the completion of process 1708,1710, 1711 to be properly placed in queue 1712 at a location ahead ofthe event of 1709 to allow the reading of the packets/events in queue1712 to occur in an expected order. Here, the order may be summarized aspackets/events occurring in the order their processes were started orevents occurred. This ordering continues for the next process shown bypoints 1713 (start of process), 1714 (process continuing), and 1715(process done).

FIGS. 18 and 19 show various systems for handling pen data in accordancewith aspects of the present invention.

Referring to FIG. 18, a pen device 1801 sends data to pen services 1803.A mouse 1802 may also generate information and send it to user32 (alsoknown as user32.dll) 1804. Some pen data (clicking on buttons forexample) may represent mouse events and is rerouted to user32 1804 to behandled as mouse events. Likewise, some mouse events may represent inkand is rerouted at 1814 to be handled as pen input. Mouse events arethen passed to the windows message pump 1805, then HWND 1806, HWND mousedevice 1812, the input queue 1810 and then input manager 1811. Pen input(for example, when the pen is moved into a range of a potential inkingservice) or a stylus event is sent to the pen input manager 1807/1808(unmanaged and managed real time stylus services). The pen input manager1807/1808 handles all the communication between the pen services 1803and the application. This handling may be performed on a normal prioritythread or on a high priority status thread. Pen services 1803 maygenerate various events. Pen events may be sent to the pen input manager1807/1808 to bypass the standard messaging system 1804-1806 and 1812.

Pen services 1803 may generate the following events: stylus in range(where the stylus has come into range of the digitizer), stylus out ofrange (where the stylus can no longer be detected by the digitizer),stylus packets (raw point data from a digitizer—the pen may or may notbe in contact with the digitizer), stylus in air points, tabletadded/removed, and system gesture events (for instance, “press and hold”and hover events).

The stylus may generate raw preview events. These raw preview events maybe handled by a real time inking service 1901 (from FIG. 19). Otherservices may listen for these events as well. The real time inkingservice 1901 may perform a number of steps on the raw preview eventsincluding:

-   -   a. Check if event is a down “Packet” event. If not stop        processing and return null.    -   b. Check if event would have been on an inking element based on        its cached layout information. If not stop processing and return        null.    -   c. Since this packet is for a “down” event, inside an inking        area, then incrementally draw the stroke.    -   d. Finally return to the pen input manager 1807/1808 the element        on which the stroke was drawn.

The return value from the raw preview event is used by the pen inputmanager 1807/1808 to “Target” the event. If a non-null target isreturned then the event may be placed on the input queue with thatspecific target. This ensures that ink that has been drawn on an elementis actually delivered to that element rather than some other element asa result of other events in the input queue.

Now the stylus event is ready to become an input event. However in mostcases, each stylus event also has a corresponding mouse message flowingthrough the message system. Before the pen input manager 1807/1808 canconvert the stylus event into an input event, it should first match upthe event with the corresponding mouse message. If necessary, the peninput manager 1807/1808 can wait for the mouse message to arrive.

Once the pen input manager 1807/1808 has both the mouse message and thestylus event, it combines the two into the appropriate input report andplaces the report on the input queue 1810.

FIG. 20 shows a queue in accordance with aspects of the presentinvention. FIG. 20 includes a queue 2001. For simplicity, the queue isshown as a circle. Alternative forms of the queue may also be usedincluding a linear queue and other versions as are known in the art.Queue 2001 includes a start pointer 2002 and end pointer 2008. Queue2001 also includes a number of locations having information (2003, 2004,and 2009). Locations 2005 and 2007 are empty. Location 2006 has beenlocked. Location 2006 may have been locked as a place holder tosynchronize events and data from different sources (an example of whichis shown in FIGS. 17A and 17B). Also, location 2006 may be locked whenthe data contained within it is being passed through a plug-incollection or chain. Locations 2010 and 2011 are empty as well. Thenumber of empty spaces may be increased or decreased as needed. Forinstance, additional spaces may be added to store additional data if thequeue is full. Alternatively, the size of the queue may be fixed suchthat any data more then can be held in the queue is discarded. This mayprovide a useful benefit to the user in that it provides an indicationto the user that the system is highly involved with other processes andit would be helpful to slowdown on the creation of new data for thequeue. The discarding of information may also show to the user that thesystem has locked up and entering additional data is not advised untilthe system continues processing again.

Application Programming Interfaces

The following provides an overview of the various applicationprogramming interfaces that may be used by aspects of the presentinvention.

The real time stylus may be part of a real time stylus input class. Theinstantiated real time stylus object may be attached to a window handleor control. The real time stylus may include default constructors. Usinga default constructor permits one to accept input only from another realtime stylus object.

To allow plug-ins to interact with a pen data stream, the real timestylus object may maintain two plug-in collections. The collections maybe specified in a property of the real time stylus object (for instance,SyncPluginCollection for synchronous plug-ins and AsyncPluginCollectionfor asynchronous plug-ins). One may add a plug-in to either collectionby calling an add method (for instance, StylusSyncPluginCollection.Addfor adding a plug-in to the synchronous collection orStyluAsyncPluginCollection.Add for adding a plug-in to the asynchronouscollection) on the appropriate property.

Synchronous plug-ins may implement an interface (for instance,IStylusSyncPlugin) and asynchronous plug-ins may implement a differentinterface (for instance, IStylusAsyncPlugin). Each plug-in may haveproperty that specifies its data interest (for instance,IStylusSyncPlugin.Datalnterest or IStylusAsyncPlugin.Datalnterest). Thereal time stylus object may call notification methods of the plug-in formethods in which the plug-in has subscribed.

The real time stylus object implements the IStylusAsyncPlugin interface.To instantiate a real time stylus that accepts input from another realtime stylus may include using a default constructor. Further alternativemechanisms may be used to associate a child or cascaded RTS with aparent. Another mechanism may use a dedicated pointer associated with aparent RTS to hold a reference to the child RTS. Here, the parent RTSwould not need to implement the asynchronous plug-in interface.

The real time stylus object may have two or more internal queues thattransport pen data. These include the input queue and the output queue.Another queue may be added between these two queues. Also, the inputqueue may be eliminated when data is only intended to enter thesynchronous plug-ins from a pen data stream. The process that thesequeues use to handle pen data is described above in relation to at leastFIGS. 6, 8, 11 and 13.

The following is an example scenario for using the real time stylusobject on a form that collects ink.

-   -   a. Create a form that implements the IStylusAsyncPlugin        interface.    -   b. Create a real time stylus object attached to a control on the        form.    -   c. Set the form to be interested in (and therefore receive)        notifications relating to a stylus contacting a surface (a        StylusDown event), data packets, and stylus up (e.g., a StylusUp        event) notifications in the form's property relating to data in        which it is interested (e.g., a DataInterest property).    -   d. In the form's IStylusAsyncPlugin.StylusDown,        IStylusAsyncPlugin.Packets, and IStylusAsyncPlugin.StylusUp        methods, add code to handle the stylus down, packets, and stylus        up notifications that are sent from the form's real time stylus        object.

Each real time stylus object maintains a list of unique identifiers forthe objects with which it can interact. The real time stylus object mayhave a few methods for translating between a unique identifier and thetablet object. These may include the GetTabletContextIDFromTablet andGetTabletFromTabletContextID methods.

The pen data may occupy its own name space (for instance,Styluslnput.PluginData namespace).

A tablet property description object (e.g., TabletPropertyDescriptionobject) may contain a property globally unique identifier (GUID) and atablet property metrics (e.g., TabletPropertyMetrics) structure thatdescribes the range, resolution, and units of the property for aspecific tablet.

A method may exist that takes a unique tablet identifier and returns acollection of property description objects supported by the tablet. Forinstance, the GetTabletPropertyDescriptionCollection method of the realtime stylus object takes the unique tablet identifier and returns thecollection of TabletPropertyDescription objects supported by the tablet.The real time stylus object's GetDesiredPacketDescription method returnsan array of GUIDs for the packet properties that the real time stylusobject will forward to its plug-ins.

The set of packet properties that is sent to the plug-ins for a giventablet may be the intersection of the set returned by theGetTabletPropertyDescriptionCollection method and the set returned by aGetDesiredPacketDescription method. To modify the set of packetproperties the real time stylus will pass to its plug-ins, the real timestylus object's SetDesiredPacket Description method may be called.

When the real time stylus object is enabled, each plug-in receives acall to its IStylusSyncPlugin.RealTimeStylusEnabled orIStylusAsyncPlugin.RealTimeStylusEnabled method. TheRealTimeStylusEnabledData object passed in the notification contains acollection of the context identifiers for the available tablets at thetime the RealTimeStylus object is enabled. When a tablet that theRealTimeStylus object can use is added to or removed from a pen-enabledcomputing system while the RealTimeStylus object is enabled, theRealTimeStylus object notifies its plug-ins that a tablet has been addedor removed.

Styluses may have additional interfaces. Each stylus may haveinformation associated with it that describes packets associated withthe styluses. The real time stylus object may pass information about thestyluses to the plug-ins in a number of notification methods.Information about a pen or stylus may be represented by a stylus object.The object is a snapshot of the state of the tablet pen at the time thedata was gathered. Since plug-ins are receiving the tablet pen data aspart of the tablet pen data stream, the plug-ins should use theinformation in the stylus object instead of checking for the currentstate of a particular tablet pen through the cursor class.

The following describes plug-ins and the RealTimeStylus class. Plug-ins,objects which implement the IStylusSyncPlugin or IStylusAsyncPlugininterface, can be added to a real time stylus object.

Both the IStylusSyncPlugin and IStylusAsyncPlugin interfaces define thesame methods. These methods allow the real time stylus object to passthe pen data to each plug-in. The IStylusSyncPlugin.DataInterest andIStylusAsyncPlugin.Datalnterest properties allow each plug-in tosubscribe to specific data in the tablet pen data stream. A plug-inshould only subscribe to the data necessary to perform its task, whichminimizes potential performance issues. Alternatively, additionalsubscriptions may be included for a plug-in.

The real time stylus object may use objects in theStylusInput.PluginData namespace to pass the pen data to its plug-ins.The real time stylus also catches exceptions thrown by plug-ins. When itdoes so, it may notify the plug-ins by calling theIStylusSyncPlugin.Error or IStylusAsyncPlugin.Error method.

The following describes plug-in data and the real time stylus class.

Plug-ins for the real time stylus may implement either theIStylusSyncPlugin or IStylusAsyncPlugin interface. One may or may notimplement all methods in the real time stylus.

The methods defined on the interfaces may use objects in theStylusInput.PluginData namespace to pass the pen data to the plug-ins.The following table describes the data objects that are parameters inthe notification methods and lists the DatalnterestMask value associatedwith the notification.

DataInterestMask Plug-in Data Value Description CustomStylusDataCustomStylusDataAdded Custom application data that is added by aplug-in. ErrorData Error Error information that the real time stylusobject adds in response to an unhandled exception in one of its plug-ins. InAirPacketsData InAirPackets Packet information for stylus motionwhile the stylus is in-air above the digitizer. InAirPacketsData PacketsPacket information for stylus motion while the stylus is touching thedigitizer. RealTimeStylusDisabledData RealTimeStylusDisabled Informationthe real time stylus object adds when it is being disabled.RealTimeStylusEnabledData RealTimeStylusEnabled Information the realtime stylus object adds when it is being enabled. StylusButtonDownDataStylusButtonDown Information about the particular stylus button that isbeing depressed. StylusButtonUpData StylusButtonUp Information about theparticular stylus button that is being released. StylusDownDataStylusDown Packet information for stylus as the stylus is brought incontact with the digitizer. StylusInRangeData StylusInRange Informationabout the particular stylus that is entering the input area of the realtime stylus object or entering the detection range of the digitizerabove the input area of the real time stylus object.StylusOutOfRangeData StylusOutOfRange Information about the particularstylus that is leaving the input area of the real time stylus object orleaving the detection range of the digitizer above the input area of thereal time stylus disabled data object. StylusUpData StylusUp Packetinformation for stylus as the stylus is lifted from the digitizer.SystemGestureData SystemGesture Information the RealTimeStylus objectadds when it detects a system gesture. TabletAddedData TabletAddedInformation about the tablet that is being added. TabletRemovedDataTabletRemoved Information about the tablet that is being removed.

The real time stylus object passes information about the tablet pen toits plug-ins in a number of the notification methods. Information aboutthe tablet pen is represented by a Stylus object. This object is asnapshot of the state of the tablet pen at the time the data wasgathered. Since plug-ins are receiving the tablet pen data as part ofthe tablet pen data stream, the plug-ins should use the information inthe Stylus object instead of checking for the current state of aparticular tablet pen through the Cursor class. Each Stylus objectcontains the tablet context identifier for the tablet that generated thedata.

The following 13 functions may be implemented on the synchronous andasynchronous interfaces. The table compares the information that ispassed through the various events. The list of events and parameters isillustrative and not exhaustive. Other events and parameters may be usedwith or in place of the following.

Many of the synchronous parameters differ from the asynchronousparameters only in that the former allows the recipient to modify thedata passing through the queue. Alternatively, both sets of parametersmay be allowed to modify the data passing through the queues.

Asynchronous Synchronous Parameter Parameter Function PurposeDescriptions Descriptions ContextCreate Used to determine when theIncludes the source of the Includes the source (a.k.a. tablet contexthas been pen (including but not of the pen (including RTSEnabled)created so that event chains limited to a pen ID), the but not limitedto a can be enabled. Consumers tablet identification and a pen ID), thetablet of this event can use this to description of the dataidentification and a know when RTS is finished packets. description ofthe with tablet initialization and data packets. is ready to beginfiring data to client code. ContextDestroy Used to determine when theIncludes the source of the Includes the source (a.k.a. tablet contexthas been pen (including but not of the pen (including RTSDisabled)destroyed so that RTS can limited to a pen ID) and but not limited to aclean up its objects. the tablet identification pen ID) and theConsumers of this event can tablet identification use this to know whenRTS is about to free a tablet context (perhaps because RTS.Enabled isfalse) and cache any PacketDescription data as needed before it is nolonger available. CursorNew Notification that a new Includes the sourceof the Includes the source stylus has been encountered pen (includingbut not of the pen (including by the digitizer limited to a pen ID) andbut not limited to a information relating to pen ID) and the stylus(including but information relating not limited to buttons on to thestylus the stylus having been (including but not activated and the like)limited to buttons on the stylus having been activated and the like)CursorInRange Notification that a stylus Includes the source of theIncludes the source has moved into range of the pen (including but notof the pen (including digitizer limited to a pen ID) and but not limitedto a information relating to pen ID) and the stylus (including butinformation relating not limited to buttons on to the stylus the stylushaving been (including but not activated and the like) limited tobuttons on the stylus having been activated and the like)CursorOutOfRange Notification that a stylus Includes the source of theIncludes the source has moved out of range of pen (including but not ofthe pen (including the digitizer limited to a pen ID) and but notlimited to a information relating to pen ID) and the stylus (includingbut information relating not limited to buttons on to the stylus thestylus having been (including but not activated and the like) limited tobuttons on the stylus having been activated and the like) CursorDownNotification that a stylus tip SourceRealTimePen, SourceRealTimePen, is“touching” the digitizer StylusInfo, StylusInfo, surfacePropertyCountPerPacket, PropertyCountPerPacket, PacketDataRef PacketDataCursorUp Notification that a stylus tip Includes the source of theIncludes the source is no longer touching the pen (including but not ofthe pen (including digitizer surface limited to a pen ID) and but notlimited to a information relating to pen ID) and the stylus (includingbut information relating not limited to buttons on to the stylus thestylus having been (including but not activated and the like). limitedto buttons on This may also include the the stylus having property countper data been activated and packet and a reference to the like). Thismay the packet data type. also include the property count per datapacket and a reference to the packet data type. InAirPacketsNotification of stylus Includes the source of the Includes the sourcemovement above the pen (including but not of the pen (includingdigitizer surface limited to a pen ID) and but not limited to ainformation relating to pen ID) and the stylus (including butinformation relating not limited to buttons on to the stylus the stylushaving been (including but not activated and the like). limited tobuttons on This may also include the the stylus having property countper data been activated and packet and a reference to the like). Thismay the packet data type. This also include the may also include a dataproperty count per packet buffer length and data packet and a the countof the data reference to the packets. packet data type. This may alsoinclude a data packet buffer length and the count of the data packets.Packets Notification of stylus Includes the source of the Includes thesource movement while touching pen (including but not of the pen(including the digitizer surface limited to a pen ID) and but notlimited to a information relating to pen ID) and the stylus (includingbut information relating not limited to buttons on to the stylus thestylus having been (including but not activated and the like). limitedto buttons on This may also include the the stylus having property countper data been activated and packet and a reference to the like). Thismay the packet data type. This also include the may also include a dataproperty count per packet buffer length and data packet and a the countof the data reference to the packets. packet data type. This may alsoinclude a data packet buffer length and the count of the data packets.SystemEvent Notification of a system Includes the source of the Includesthe source event or gesture. Examples pen (including but not of the pen(including include “tap”, “double tap”, limited to a pen ID) and but notlimited to a “press & hold”, “shake” information relating to pen ID) andthe stylus (including but information relating not limited to buttons onto the stylus the stylus having been (including but not activated andthe like). limited to buttons on This may also include the stylus havingsystem event information been activated and and data relating to the thelike). This may system event. also include system event information anddata relating to the system event. TabletAdded Notification of a newIncludes the source of the Includes the source digitizer connected tothe pen (including but not of the pen (including system (typically anlimited to a pen ID) and but not limited to a external USB device)information relating to pen ID) and the tablet information relating tothe tablet TabletRemoved Notification of a digitizer Includes the sourceof the Includes the source disconnected from the pen (including but notof the pen (including system (typically an limited to a pen ID) and butnot limited to a external USB device) information relating to pen ID)and the tablet that was information relating removed to the tablet thatwas removed UserData Arbitrary user data, n/a Includes the sourceuniquely identified (for of the pen (including instance, by a GUID).This but not limited to a is used by a specialized pen ID), a GUIDplug-ins which wants to associated with the pass information user'sdata, an downstream to the Ink indication of the size Collection Objectthrough of the user's data, the queue and guarantee that and the contentof it comes in the right order the user's data with respect to the restof the real-time data described above.

Aspects of the present invention have been described in terms ofillustrative embodiments thereof. Numerous other embodiments,modifications and variations within the scope and spirit of the appendedclaims will occur to persons of ordinary skill in the art from a reviewof this disclosure.

1. A computing system that orders events from a keyboard, mouse, andstylus comprising: a storage; a processor exchanging information withsaid storage, said processor hosting a pen services component, said penservices receiving information from said mouse and said stylus, orderingsaid information, and forwarding information related to electronic inkto a pen input manager; a first input receiving information from saidmouse; and a second input receiving information from said stylus,wherein the pen input manager includes a dynamic renderer and a staticrenderer, wherein said pen input manager includes an unmanaged portionand a managed portion.
 2. A system for ordering inputs from a stylus anda mouse comprising: a processor supporting components, said thecomponents comprising: a windows message pump that receives input from auser; a pen input manager that receives stylus input from said windowsmessage pump; an HWND stylus device that outputs said stylus input to aninput queue; a real time inking service attached to said pen inputmanager, said real time inking service processing said stylus input,said real time inking service outputting said stylus input to a dynamicrenderer and a static renderer; said processor including: a HWNDcomponent that receives mouse input from said windows message pump; aHWND mouse component that receives output from said HWND component andforwards said mouse input to said input queue. wherein said input queuealso receives mouse inputs, wherein said pen input manager componentincludes an unmanaged portion and a managed portion, wherein said realtime inking service is attached to said managed portion of said peninput manager.
 3. A method for handling stylus input and mouse inputcomprising the steps of: receiving at a windows message pump said stylusinput and said mouse input; sending said stylus input to a pen inputmanager; sending said mouse input to a HWND device manager; exchangingsaid stylus input with a real time inking service having a dynamicrenderer and a static renderer; outputting information from said realtime inking service to a HWND stylus device; outputting information fromsaid HWND stylus device to an input queue; and receiving in said inputqueue information relating to said stylus input and information relatingto said mouse input; and outputting information relating to said stylusinput and said mouse input from said input queue to an input manager,wherein said information output from said HWND stylus device is saidinformation relating to said stylus input.